Mapshaper is a free and open-source tool that is best known for fast and easy simplification. Other tools for simplification – like QGIS or ogr2ogr – do not preserve topology while simplifying. This means you may get sliver polygons or missing intersections. Mapshaper performs topologically-aware simplification and gives you much more control on the process.Continue reading
GDAL and OGR libraries come with handy command-line tools. These tools are quite powerful and can save you a lot of effort if you know how to use them. Here I will show how to use the ogrinfo and ogr2ogr tools to perform spatial joins. A single command can do complex operations on your spatial data and save you a lot of clicking-around and data-munging in a GIS.
Get the Tools
Get the Data
Review the data and problem statement from the Performing Spatial Joins tutorial. Download the Borough Boundaries and Nursing Homes shapefiles.
OGR command line tools accept only 1 input. But we have 2 inputs for the spatial join. An easy way to fix this, is to use a VRT file. A VRT file allows us to specify multiple inputs and pass them to the command-line tool as layers of a single input.
Unzip the input shapefiles in a single folder on your drive. Create a file named input.vrt in the same folder with the following content.
<OGRVRTDataSource> <OGRVRTLayer name="boroughs"> <SrcDataSource>nybb.shp</SrcDataSource> <SrcLayer>nybb</SrcLayer> </OGRVRTLayer> <OGRVRTLayer name="nursinghomes"> <SrcDataSource>OEM_NursingHomes_001.shp</SrcDataSource> <SrcLayer>OEM_NursingHomes_001</SrcLayer> </OGRVRTLayer> </OGRVRTDataSource>
Open the OSGeo4W shell and cd to the directory containing the shapefiles and the vrt file. Run the ogrinfo command to check if the VRT file is correct.
OGR tools can run SQL queries on the input layers. We will use the ST_INTERSECTS function to find all nursing homes that intersect the boundary of a borough and use the SUM function to find the total nursing home capacity of a borough. Run the following command.
ogrinfo -sql "SELECT b.BoroName, sum(n.Capacity) as total_capacity from boroughs b, nursinghomes n WHERE ST_INTERSECTS(b.geometry, n.geometry) group by b.BoroName" -dialect SQLITE input.vrt
You can see that in a single command we got the results by doing a spatial join that takes a lot of clicking around in a GIS environment. We can do a reverse spatial join as well. We can join the name of the Borough to each feature of the Nursing Homes layer. Using the ogr2ogr tool we can write out a shapefile from the resulting join. Note that we are adding a geometry column in the SELECT statement which results in a spatial output. Run the following command:
ogr2ogr -sql "SELECT n.Name, n.Capacity, n.geometry, b.BoroName from boroughs b, nursinghomes n WHERE ST_INTERSECTS(b.geometry, n.geometry)" -dialect SQLITE output.shp input.vrt
Open the output.shp in a GIS to verify that the new shapefile as attributes joined from the intersecting borough. You can use ogrinfo command to check that as well.
ogrinfo -al output.shp
The Association for People with Disability (APD) is a non-profit organization based out of Bangalore, India. Their mission is to reach out and rehabilitate people with disability from the under privileged segment. Over the past year, I along with my colleagues have been volunteering with them to develop a system that can help improve their field data collection efforts.
APD provides variety of services rehabilitate under privileged people with disabilities. Before they can render any service, they need register the individuals with the organization and collect basic background information. Registrations are mostly done in their field offices or at camps organized in rural areas. Paper forms were filled at the site and shipped to their field office. A staff member entered the data manually into their software platform.
This has many problems:
- The text on the paper form was often illegible. Some fields were missing or inaccurate.
- Data entry was laborious and introduced errors.
- 4–6 weeks of lag time before the data was available in the system.
We helped APD implement a process using OpenDataKit (ODK) that allowed capture of the form using android devices. With the new system, the data is captured on the mobile device using the ODK Collect app in the field and sent to a ODK Aggregate server running on Google AppEngine. The data ends up in a shared spreadsheet which is imported to APD’s system after each registration camp.
This new workflow offers several advantages over the paper based forms:
- Reliable data collection in areas with poor network connectivity. ODK Collect app can work completely offline and the data is stored on the device memory. Once the staff members are back in office and connect to a WiFi network — the data is sent to the server.
- The mobile app enforces checks, so all the data is consistent and there are no missing fields.
- Allows for the capture of pictures and additional metadata (such as time, location, staff id).
- The data is exported from the spreadsheet and imported to their system the next day — cutting the lag from weeks to hours.
- October 2014: Met with the APD registration team to understand their requirements and design a process.
- November 2014: A prototype is created by migrating the registration form to ODK using XLSForm. APD trains field staff on using the mobile app. Successful field test.
- December 2014: First full-fledged camp with 3 devices. Successful registration of 55 participants. Staff is very happy with the increased speed, accuracy and reduction in delay in getting the registrations processed.
- January 2014: APD moves their registrations completely to mobile devices. All registrations are completely paperless and have the added benefit of having participant’s picture as part of the registration process. Over 500 registrations processed without a problem.
While this is a great start, we are looking at helping them with other challenges in the field. In the coming months, we want to tackle the following problems:
- Migrate patient visit and treatment forms to OpenDataKit. These require having access to the patient’s medical history in the field. OpenDataKit’s 2.0 suite of tools would be a good fit.
- Task allocation and scheduling optimization for the field staff.
- Encoding the knowledge from the training manual to a mobile app.
A long pending weekend project is done. Printed, cut and folded a sturdy globe using the template from Le Paper Globe.
This is not only fun, but a good prop to learn more about Geography. I envision it would make a fun do-it-yourself project with kids of all ages.
I recently had a need to calculate distance between a large number of latitude/longitude coordinate pairs. There are many options available if you want to import these in a GIS and run analysis. But there is a simpler and much more accesible way if you aren’t doing very high accuracy calculations.Continue reading
I was at #cartonama conference in Bangalore on 23rd September, 2012. It was wonderful to be among geogeeks and see the energy and enthusiasm. Got to see some great demos and LBS apps from startups in India. Here’s a run down from my notes and bookmarks.
- Traffic cams in Bangalore: This was probably the coolest demo for me. Love the simplicity and usefulness. Just open this URL in your smartphone browser and get real-time feed from the traffic-cam neareast to you from BTIS.http://hackday.freecrow.org/trafficam
- Thru the Gall: A Layar-based android app to solve the last-mile direcction problem in India. Bubbles guide you through the narrow-lanes (galli). http://www.layar.com/layers/galli/
- Padma: Search and Browse Indian documentaries through locations of the clips in the documentaries http://pad.ma/map
- DelightCircle: LBS app to discover offers and discounts near you. Surprised at the coverage and partnerships they are able to genrate in the short time. Powered by MongoDB at the backend.http://delightcircle.com/
- Commonfloor.com: Google Maps API app for property search. Nicely done and simple UI and fast lad times. The team talked about challenges and techniques they use to de-dup the listings and pin the individual listings to the apartment block. http://www.commonfloor.com/apartments-for-sale/maps/
- Yahoo: I had forgotten about yahoo as a geo player. But from the talk the team seems pretty excited about their geo APIs. The Geo APIs are used by many yahoo properties internally and available to developers too. The API to extract locations from free-form-text was cool. YQL is a SQL-like query interface for their API. http://developer.yahoo.com/yql/console/, APIS at http://developer.yahoo.com/geo/
- NextDrop: Very interesting idea. Most places in India do not have a regular water supply and people’s day revolves around when they are able to collecct water. They have built a system that alerts subscribers by sms when they can expect water based on the information from the utility company. Subscribers pay Rs.10/month to get this data. Their challenge revoled around locating their customers on a map – both for billing as well as determining which pipeline will supply them water. http://nextdrop.org
- Lokasi: Simple android app that allows you to share your current location by sms. Reverse-geocoding + location identifier. http://www.onze.in/ . collected POIs + street geometry for entire bangalore is about 6 months with a 10-person team.
- Brief history of map making: Fun talk with lot of pretty and interesting maps. No geo conference is complete without a passionate discussion on projections 🙂 http://www.slideshare.net/sumandro/sumandromapping-thecity27032012-12203410
- Chalo BEST: Bus route planner for Mumbai using OpenTripPlanner and GTFS feed from BEST. http://trip.chalobest.in/. Challenges in converting messy spreadsheet data from the agengy in to GTFS feed.
- Geohash: overview of the geohash system. Lots of discussion around what geohash is good for. http://geohash.org/
India’s first FOSS4G – India conference was organized on 25-26 October, 2012 at Hyderabad. This was a surprisingly low-profile event and there was very little buzz about it on the internet. I learnt about it just in time to register and make it to the event. Here I am sharing some notes and impressions. (notes are mostly from memory, so please excuse if I missed something)
- I haven’t been to other FOSS4G events, but this was unlike any other Geo conference I have attended. It was very formal and government-centric. All presentations were from government researchers, employees or students.
- The penetration of open-source software within government was deeper than I anticipated and the decision-makers – including the politicians recognize the benefits of using open technologies. The Chief Guest – Shri V. Aruna Kumar, MP from Rajahmundry district commented that he realizes that the real cost of software is not the upfront cost, but the lifetime maintenance cost and open-source and open-standards can really benefit in the longer run.
- Keynote’s main message was that we should really focus on delivering applications that solve real problems. According to the speaker – Shri Anoop Singh, Special Secretary to AP Government – we have not even begun to realize the full potential of GIS within governments. For most end-users and decision makers, the ‘Science’ or ‘System’ in GIS isn’t important and GIS should really be about Geographic Information Services.
- India is seen largely as a follower in the open source movement and no big open-source projects have come out of research institutions here. To address this, International Institute of Information Technology released one of their projects – LSIViewer as open-source. The github upload was done during the inauguration of the conference by the chief guest!
- Prof. Venkatesh Raghavan receiving 2012 Sol Katz award Prof. Venkatesh Raghavan, Osaka City University was chosen as the winner of the 2012 Sol Katz ward by OsGeo. The award was given to him during the conference.
- Postgrasql + PostGIS, Geoserver and Openlayers seem to be the platform of choice. Almost all applications I saw were built using one of these.
- ELOGeo portal was shown an example of learning resources available for open source geo tools.
- There is virtually no open data for India given the strict geo data policy. But even within the government departments – there is a lot of reluctance in data sharing. Everyone talked about this problem of data being guarded very closely and seldom shared. But some forward thinking departments, especially in Tamil Nadu, have found a way around it by promoting Web Services. They publish a WMS feed for the datasets. That way other departments can use the data without really having any ownership of it. This seems to be working well.
- Ms. Mahalakshmi from National Informatics Center (NIC) in Tamil Nadu demonstrated some of most interesting internal applications built by them. They have built web services for different departments and each department uses these web services for their application. Administrative boundaries, census data, land use maps, land parcels, police data and planning related data are some examples which are available via WMS. She also showed a pretty cool app that BSNL – Chennai uses to identify fault points in their 2G network. The map is updated in real-time with pings from their different 2G sites around Chennai.
- BHUVAN – Indian Space Research Organization’s (ISRO) portal for disseminating geodata has come a long way since it’s launch a few years back. It’s built completely using open-source stack and it’s a pretty snappy app. For the first time, a lot of thematic datasets are available freely to the public via WMS. They are also making some medium and low resolution raw/derived data from Indian remote sensing satellites available for download. The downloads are for personal use only, but this is a big step forward in India’s data access policy which has been very restrictive so far.
In summary, some positives and challenges for Open-source software described by various speakers.
- No upfront cost. Don’t have to navigate complicated and time-consuming approval and procurement process. Easy to get started at low-cost and do a proof-of-concept.
- Standards Compliant. Rising awareness within the government of being standards compliant. Choosing a proprietary solution which is not standards compliant can cause problems and raise questions from others in future.
- No vendor lock-in. Can hire another consultant to run or develop services since all code is available.
- Lack of support. Not many companies providing development and support. Most applications shown were written and maintained by in-house staff. Big opportunities for consultants who can provide support for open source tools.
- Change aversion: On the desktop, most people are used to proprietary software from their education or prior work. One example given was reluctance in the government to switch to Open Office. Free versions of MS Office is easy to come by and no incentive to learn and adopt open source solution. Using open-source software in education is a way to increase the adoption.
- Lack of Mature solutions: Politicians and administrators felt they had limited time in their tenure to implement whatever changes they want and show progress. There is always the attraction to choose a well known solution and get the results. The key message was that they feel the open-source software is mature but there are no mature solutions based on these that the governments can adopt without much risk.
Many students have asked me for ideas on what topic they should choose for their Thesis. I have debated this myself when I was a student. The ideal topic would be the one that allows you to dive into a topic deeply as well as give you some practical skills that will help you landing a job. Here are some pointer that may help you make that decision.
A mashup is created by merging 2 or more sources of location data. There are variety of free data available for you to experiment with and create unique applications. Map mashups can be created using free tools such as Google Maps, OpenLayers, GeoCommons Maker!
There is always a need to implement new features for open-source GIS software like QGIS or MapWindowGIS. You can implement a cool feature or customize the software for a particular domain. You will not only gain valuable experience developing among the best programmers but also this will be an impressive addition to your resume.
Most often students struggle to find good quality free GIS data which can be used in interesting ways. Contact local NGOs to see if you can help them analyze their location-specific data. Most NGOs collect some form of loation data and your GIS skills can come in handy to them. The result of such a project could be a unique GIS project that uses real on-the-ground data.
If you are at a university, a good project would be to put together a web-accessible campus information system. You could model the campus buildings in 3D, use satellite imagery via an API and overlay data about your campus. Such a system could be attractive to the administrator as well.
You can buy a cheap GPS receiver and implement a vehicle tracking system. This will get you exposed to GPS/GSM protocols and get you hands-on knowledge that many companies worldwide are looking for. Add your own twist to the traditional tracking system by specializing it for walking directions, bus travel, toursut sightseeing, navigating the blind etc.
Everyone is turning to green and sustainable technologies. As an intern or a researcher, you could use your analytical skills to further the knowledge in this area. Wind power, solar energy, climate change etc. are great topics for GIS and Remote Sensing research.Analysing wind data to find best locations for wind turbines or using remote sensing images to detect climate change are some examples of projects. If you are a PhD student, you’ll find these topics complex and challenging enough for yor dissertation and you’ll stand to recieve grants or funds from governments as well.
Do you have more ideas to share? What project are you thinking of? Please leave a comment and help the GIS student community!