Databases Basics and Spacial Matrix - Discussig Geographic Potentials of Databases.
•Download as PPTX, PDF•
0 likes•3 views
A core introduction to Data Types, Databases around us and the use cases, Integrating GeoSpacial Arrangements to our projects. This include Speaker Notes. and is indented to give an overall idea about things not focusing on a specific focus. This Incudes references of Relational Databases likePostgresql and Mysql, Key Value databases like Redis, Document DB like MongoDB, Search Engines like ElasticSearch, and the second part handles with GeoSpatial Arrangements in Postgis.
Recommended
Recommended
More Related Content
Similar to Databases Basics and Spacial Matrix - Discussig Geographic Potentials of Databases.
Similar to Databases Basics and Spacial Matrix - Discussig Geographic Potentials of Databases. (20)
More from Jerin John
More from Jerin John (6)
Recently uploaded
Recently uploaded (20)
Databases Basics and Spacial Matrix - Discussig Geographic Potentials of Databases.
- 1. Databases and Spatial References Discussing Geographic Potentials
- 2. We Discuss On: - Different Databases around us (exp. 10 min) - Data Types - Purpose and Examples - Geospatial Arrangement of Data (exp. 15 mins) - Spatial Data Types - SQL - PostGIS - SRID and other Standards - Supporting Databases
- 3. Different Databases around us Before that, Assume you are a developer at linkedin, and You are assigned to share a user’s post to all his 2nd and third degree connections in a geographical area.! What will be your “data structure” and “query” to find those users?
- 4. Data Types VARCHAR INT DATE ENUM (predefined constants) TEXT FLOAT TIME BLOB (file objects) CHAR BOOLEAN DATETIME JSON / XML CITEXT (case insensitive text) DECIMAL TIMESTAMP SET / ARRAY UUID DOUBLE YEAR BIT As you all know……
- 6. Relational DB - As you know, Tabulated Storage Structure - Support Relations - Reputed for Structured Datas like ecommerces, B2B e.t.c. Examples Descriptions PostgreSQL Open Source | Stable | Reliable | read- intensive | heavy write loads for Medium-sized databases MySQL Open Source | fast | Lightweight | read- intensive for Small-sized databases Sqlite Development Friendly | File based | Low memory devices
- 7. Key Value DB (NoSQL) - Supports Storing Data against hashed Keys - Quick Access - Mainly used for Caching Examples Descriptions Redis In-memory storage, persistent on disk, supports various data structures, replication, high availability Etcd Distributed reliable key-value store, ideal for configuration and service discovery with strong consistency guarantees Amazon DynamoDB Managed NoSQL database, key-value and document data models, single-digit millisecond performance at any scale.
- 8. Graph DB - What will be the best structure to share a linkedin post in 3rd level connection? - Stores Relations in Graph. Examples Descriptions Neo4j Relationships-focused, flexible schema, Cypher query language, ACID transactions, high-performance graph queries. JanusGraph Scalable graph database optimized for storing and querying large graphs with billions of vertices and edges. ArangoDB Multi-model database, supports graph, document, and key/value data models in one database core.
- 9. “Initial” Google Maps. 4 KM 3 KM 4 KM 4 KM
- 10. DocumentDB (NoSQL) - Supports Documents for unstructured Datas - Support Relations - Reputed for Structured Datas like Blogs, Versioned Datas, Scraping, Storing Catalogues. Examples Descriptions MongoDB Dynamic schema, rich queries, high performance, replication, horizontal scaling. CouchDB JSON-based document storage, RESTful HTTP API, map-reduce views, and replication features Amazon DynamoDB Fully managed, multi-region, durable with built- in security, backup and restore, and in-memory caching
- 11. Search Database - Mainly Used as potential Search Engines. - Capable of filtering searching and find complex analytics. - Mostly Memory Based Examples Descriptions Elastic Search Full-text search, analytical queries, scalable, distributed nature, real-time indexing. Apache Solr Open-source search platform, powerful full- text search, hit highlighting. Sphinx Full-text search engine, provides text search functionality to client applications.
- 12. Flipkart Filter Page https://blog.flipkart.tech/
- 13. Time Series DB - Supports Storing Data against hashed Keys - Quick Access - Mainly used for Caching Examples Descriptions InfluxDB High write loads, time-stamped data, downsampling, retention policies, real-time analysis. TimescaleDB Open-source time-series SQL database optimized for fast ingest and complex queries. Prometheus Open-source monitoring system with a dimensional data model, flexible query language, and autonomous server nodes.
- 14. Geospatial Arrangement of Data Let’s Take a small problem; , Assume you are a developer at Airbnb, and You are assigned to locate nearby hotels from my location. What will be your query?
- 15. - Everyone know about location coordinates (latitude, longitude, topology). - Collectively known as Geographic Information System (GIS). - And the term `spatial reference`. - Different organization follows different standards such as what-three-words for “spatial reference”, - Most popular standard is Key Points - SRID 4326 - World Geodetic System 1984 (WGS 84) and 3857 - Web Mercator.
- 16. SRID defines multiple standards which defines: - the metrics to be used (like inc feet cm) - what all parameters have to be considered (lat, long, topology, sealevel, ) What is SRID (Spatial Reference System Identifier).
- 17. Implementation
- 18. POINT POINT(97.1234 10.1222) Represents a single point on the Earth's surface defined by its longitude and latitude coordinates. LINESTRING LINESTRING(0 0, 1 1, 2 2) Represents a sequence of connected line segments defined by coordinates of its vertices. POLYGON POLYGON((0 0, 1 0, 1 1, 0 1, 0 0)) Represents a closed shape with an arbitrary number of vertices, forming a filled area on the map. GEOMETRY GEOMETRY(POINT, 4326) Represents any geometric object. MULTIPOINT MULTIPOINT((0 0), (1 2), (2 3)) Represents a collection of points. MULTILINESTRING MULTILINESTRING( (0 0, 1 1), (2 2, 3 3)) Represents a collection of LineStrings. MULTIPOLYGON MULTIPOLYGON( ((0 0, 1 0, 1 1, 0 1, 0 0)), ((2 2, 3 2, 3 3, 2 3, 2 2))) Represents a collection of Polygons. GEOMETRYCOLLECTIO N GEOMETRYCOLLECTION( POINT(1 2), LINESTRING(1 2, 3 4) ) Represents a collection of other geometries.
- 19. Saving data into GeoLocation Field ● Let’s create a table hotel with its geolocation; ● Insert some data into it.
- 20. Querying With Distance Searching , for 3 nearest hotels which is within 5000 meters radius from my location [POINT(76.2673 9.9312)].
- 21. Querying Within a Custom Polygon Drawing a polygon and finding hotels inside it.
- 22. Features Provided Spatial Queries 1. Proximity Searches: Find features within a specified distance from a point, line, or polygon (e.g., finding hotels within 5 km of a location). 2. Point-in-Polygon: Determine whether a point lies within a given polygon (e.g., identifying whether a specific location falls within a city boundary). 3. Intersection: Identify where two geometries intersect, producing a geometry of the intersection (e.g., finding where a road crosses a river). Spatial Measurements 1. Distance Calculation: Measure the distance between two points, lines, or polygons. 2. Area Calculation: Compute the area of polygons. 3. Length Calculation: Determine the length of a line or the perimeter of a polygon.
- 23. Features Provided Spatial Analysis 1. Buffering: Create a buffer area around a point, line, or polygon at a specified distance (e.g., creating a 10 km buffer zone around a park). 2. Overlay Analysis: Combine two or more datasets to create a new set of geometries and attributes based on their spatial relationship (e.g., overlaying land use and flood zones to identify at-risk areas). 3. Spatial Join: Linking records from two sets of spatial data based on their spatial relationship (e.g., assigning census data to electoral districts based on location). Spatial Aggregation 1. Grouping by Area: Aggregate data based on spatial boundaries (e.g., summing population by districts). 2. Spatial Binning: Aggregate points into larger areas or bins for analysis (e.g., spacial heatmap for density analysis).
- 24. Features Provided Data Management 1. Geocoding: Converting addresses into geographic coordinates. 2. Reverse Geocoding: Converting geographic coordinates into readable addresses or place names. 3. Data Import/Export: Importing and exporting geospatial data in various formats (e.g., Shapefile, GeoJSON, KML). Indexing and Optimization 1. Spatial Indexing: Creating spatial indexes to improve the performance of spatial queries. 2. Partitioning: Dividing large datasets into smaller, more manageable pieces based on spatial criteria.
- 25. Features Provided Visualization 1. Mapping: Creating visual representations of geospatial data on a map. 2. Thematic Mapping: Displaying data based on themes or categories, such as heat maps, choropleth maps, or symbol maps. Modification and Editing 1. Geometry Editing: Altering the shape or position of geometric features (e.g., moving a point, editing a polygon's boundary). 2. Topology Editing: Managing spatial relationships between features (e.g., ensuring road networks connect without overlaps or gaps).
- 26. Geospatial support for Other Databases Database Description Extension (if needed) PostgreSQL As PostGIS extension PostGIS MySQL Spatial Data Types and Functions built-in SQLite Spatial Data Support through extension SpatiaLite MongoDB Geospatial Indexing and Queries built-in Neo4j Spatial Types and Functions for Geocoding and Routing built-in Elasticsearch Geo-point and geo-shape data types for geographic search built-in Redis Geospatial indexing and querying Redis Geo (built-in module) InfluxDB Geotemporal data support via Flux built-in
- 27. Thank You! (For No Questioning 😜 )
Editor's Notes
- So, Good Evening all, So let’s dive directly into databases. Many of the faces are relatively new to me. So I will just introduce myself.
- So We divide the presentation into two horizons. Different Databases around us and A quick but boring introduction to GeoSpatial Arrangement of Data. - means data types while handling latitude and longitudes, distances, areas and lines, and Querying in them.
- Before that, let’s put a question for the developers here: “ Assume you are a developer at LinkedIN, (come on yaar, we don’t need to pay tax for that, at least now) and you are assigned to share a Tim Cook’s post to all his 3rd and 4th degree connections in his geographical area! What will be your “data structure” and “query” to find those users? - This is the problem we are trying to tackle here. --- So before that, as Developer, what is our core responsibility?? To manage soft copy of data, Right? I hope so! Do someone have any other opinion? (Yeah! I respect that as well.) When the world is running, our big 3 headaches starts with ‘V’. They are: Volume, Velocity and Variety of data. So in order to control them, we don’t have a magic spell like Harry Potter uses. So we need engineering. The Primary Step will be classifying and storing different type of datas into different formats, Where we can perform operations on them. So There we forms … Next: Common Data Types
- … Data Types. So, if we need to operate it efficiently; as you know, we need them to be in an optimal structure; at least this topic will be boring for some of us. These are Boolean to store true or false, we can store dates times, numbers, files, and majorly text datas. These things are self explanatory. So we can skip to next topic, How Datas are stored in Relational DB. Next: How data is stored in Relational DB.
- Hope everyone is aware of term “Relational Database”. For those haven’t, Here is the most simplified definition: - This is an application, something like photoshop, google chrome, but primary purpose is to get connected by other programs, to store, modify and retrieve data efficiently. You can see a sample table above and a csv in the right side. We can imagine our data we insert into database will be stored in such a manner. --- So for those who already have an idea, Have you ever wondered, how these datas has been stored in db? This is out of our scope, yet in one line, assume for the left dataset, We have | title - max 70 chars | qty - 4 chars (smallint) | area name - maybe max 20 chars | bay-name - may be 6 chars Net it will take around 100 chars space for each row. So regardless of data is there or not, the space is holded there and can access a data, with some simple calculations, if we know the starting address of table and size of any row and delta of space for each column. Form the row. Now you can think of select * from table where areaname=’Warehouse’; can be executed over your custom script. Next Slide: This is known as Relational DB!
- And Now, This is self explanatory! As usual, A tabular storage of data. So, Relational DB is one of the advanced database techniques from 1970s. And is really useful when we have a structured data, or when we want to have a clean data in structured form! I'm Just skipping this slide as you will be familiar with this type of databases.. This brings us to the question,... do we really need a structure to store data??? Well, its upto us. And here we have another type of database to hep us, which is NOSQL Database!
- I know, at least few of you expected an example of MongoDB in this slide! Just Kidding! But I put this intentionally, to change the perspective. And Of Course! MongoDB is a good example! Here we are discussing about key value databases, which holds a is a major share of nosql database. So again, what is Database?, Databases are some applications or “files” which helps us to store and retrieve data “in-its-ease”. This can be persistent or might have some other purpose some times. When Relational database have a Structured Querying Language to fetch data, No SQL’s do not tends to follow a tabulated storage structure and they have their own querying pattern. Here the example shows how to interact with Redis. I know many of you might be familiar with Redis as a cache application or some as in memory db ? Redis focuses on performance and store data in memory. So what’s the benefit? Fetching from memory will be around at least 6 times faster than DB Storage. Retrieval of data, which makes it extremely efficient in non persistent datasets. Let’s Dive into some usages other than caching: Twitter uses redis to create your timeline - Twitter expects around 3 lakhs read per second and 6K writes per second. Instead of creating a timeline on my request, Twitter handles it in more efficient way. When each of my friend tweets, twitter will add that post id into my timeline, which is getting constructed in redis against my user id. So when I enter to redis, So on my request, Pre constructed post just have to get loaded. OK COOL! Next: So Do you remember our first question about LinkedIN Post? Let’s partially solve it next..
- Here Comes, Graph DataBases! When a Relational DB uses tables to store data, and Redis stores data into a location mapped with a key, A Graph Database uses something called "nodes" and "edges" to represent data and relationships between them. Neo4j is one of the prominent player in this segment. You can see some some Neo4J Queries to create 3 users and define a relation “FRIEND” between u1-u2, u2-u3 and u3-u1. If we ignore the case of u3-u1, u2 is the first level friend of u1 and u3 is the second level friend of u2. It brings up a mapping, u3 is one of the 2nd degree connection for u1. Thus a Graph DB can efficiently store and retrieve data using relations. Now let’s just imagine how this can be implemented in Google Maps “in its initial days”.
- When we go to depth, we can store the datas: kakkanad, palarivattom, vyttila and Palachuvadu with distance between them as weight in the relation distance, and time between them as the relation time. Kakkanad x Palarivattom (4 kms) Kakkanad x Padamugal (3 kms) Palarivattom x Vyttila (4 kms) Padamugal x Vyttila (4 kms) Now we can track all possible ways between kakkanad and vyttila. (two ways will be via palarivattom and via vyttila.) (Of Course, google have much efficient way, but for an MVP, this will be enough.) USAGE EXAMPLE: As we discussed first, Making user networks in social medias like LinkedIN / Twitter. Store Places to generate navigation in google maps / System.
- This is one of the less efficient Databases, for its features, Its my personal opinion, but horizontally scalable database to hold enormous volume of data. Explain the Usage of Amazon DynamoDB at Gateway Aggregator Project. We have more variants in it like column Databases, One of the prominent is HBase, Database created directly over Hadoop in Big Data Network. I feel lucky to work on even that. USAGE EXAMPLE: - Store Chat History (You can use a unique key for chat room, to access all the chats).
- Search Database are in mostly :in-memory: databases without promising much persistence, but provide extremely high efficiency, potentially in filtering searching, ordering and complex analytics. USAGE EXAMPLE: Product Search in Ecommerce AutoSuggestion. Summary Calculation. Live Analytics I brought you an interesting case study here: Let’s enlarge the screen shot of flipkart in the top corner.
- Sorry, If you look into this today, you will not be able to find it, flipkart has discontinued this feature, due to extreme load. I believe they have performed a good cost cutting by eliminating this minute feature. Maybe this feature isn’t performing well in their ROI conversion matrix. I took it from an old version of flipkart.com at Web Archives. > Each and everything starting from color of a button in Flipkart Tech is a chapter for another presentation, and extremely recommendable for UIUX engineers, Developers, BA and Tech leads. You can read more at blog.flipkart.tech Back to our topic. When you choose a category these things dynamically updates product count in each filtering. I have personally computed them efficiently using elasticsearch for a project and these are amazingly interesting. But if we proceed to use a non matching db like Relational for this project, they will kill the business itself. We are eating up our own time, so let’s sweep into next slide.
- Time Series Databases. These store datas against time as keys and capable to handle huge writes. In the example, you can see “Prometheus”!! I guess Devops Team knows it! In one of the previous projects, We used it to store and analyze K8S pods log in a fintech domain; So that we can analyze errors and api specific logs and much more with simple UI and queries; Before proceeding; Just discuss read intensive and write intensive databases; So There are much more; Which will be out of our scope! Column DB is one of them! USAGE EXAMPLE : Server logs, hardware / IOT logs e.t.c. It’s been so long in the first part, and now we moving on to the second part of the slide. Anyone who have lost the bus in between, it’s the last stop for you to re-enter.
- So coming to Geospatial Arrangement of data. Let’s again take a small problem; assume you are a developer at Airbnb or Booking.com, and You are assigned to locate nearby hotels from my location. What will be your query?
- So before we start, we need to be in the same page on some information. In order to point out a particular location in map, we need a Geographic Information System. We use latitude and longitude for them. This is a standard. And this standard is known as SRID 4326. It plans the earth surface into a mesh of horizontal and vertical lines named (latitudes and longitudes). Another popular one is “what-three-words”; it do not even uses latitude and longitude. This uses words.
- And here is some information about srid and its standardization. I can share the slide if someone wants. Let’s skip this theoryfor now.
- So coming to technology, as we discussed in the above slides, we have different data types to store each type of data. And now we unlocked some new data types. We want to mark POINTS, LINES, OR A POLYGON in a map. And we need to store them in Our DATABASES.
- So here are some common data types related to GEOLOGICAL INFORMATION. Again POINT, LINESTRING and POLYGON are self explanatory. GEOMETRY is like a data type which holds any of the above data types. Additionally against each of them, we may require to plot multiple points or multiple lines as a part of same field, such as a road or street lights in a street. For that we also have some additional data types to store a collection of data..
- So this is how we create a table, with a geolocation point which stores a point. And while inserting data, we can simply pass oru point as string and our database utility functions will convert them for us. I know these are a bit complex. Just understand the working rather than trying to read complex function names. Function names can be referred always.
- Here is a query to search for 3 nearest hotels which is within 5000 meters radius from my location [POINT(76.2673 9.9312)]. This also returns a air way distance between my point and that hotel’s location in meters.
- And we can even use some polygons to find all hotels inside it. For that we can pass the polygon and the location field with a contains function.