For the past seven years, faculty D’Arcy Smith and his Computer Systems Technology (CST) Diploma program students have been working on ODEN – Open Data Developer Network, which attempts to standardize Open Data and offers new possibilities to developers, data scientists, cities, and anyone wanting to use accessible data.
“Open Data (OD) is big business,” says D’Arcy. “According to research, OD has the potential to help create trillions of dollars in economic development. OD can, for example, break down information gaps across industries, allowing companies to share benchmarks and spread best practices that raise productivity and propel innovation.”
“Standardizing and cleaning data reportedly consumes approximately 80% of developers’ time”
However, there’s one major issue, says D’Arcy, and that’s the lack of standardization. “The demanding process of standardizing and cleaning data reportedly consumes approximately 80% of developers’ time before they can effectively use it. Developers need an efficient and unified platform that enables the easy discovery of OD across cities, presented in a developer-friendly format.”
And that’s where ODEN comes in.
ODEN’s secret sauce: its Transmogrifier
Once launched, ODEN will be a centralized hub and community where collaborative efforts standardize OD. The standardization aims to remove the cumbersome manual data cleaning, enabling developers to dedicate more time and resources to their core application features and data analysis.
The application’s secret sauce is its Transmogrifier, an automated tool that, according to D’Arcy, revolutionizes data extraction, transformation, and loading. Its versatility allows it to extract data from various sources, support multiple transformations, and effortlessly load the transformed data into different destinations.
“We found that each city used a different format when gathering the data”
D’Arcy gives the example of a developer who creates an app that uses data from Strava, a popular fitness tracking platform, in conjunction with air pollution data. “In Birmingham, for instance, they could use ODEN to access standardized pollution data and correlate it with Strava data to advise users on the best times and routes for outdoor activities, such as cycling or running, based on air quality. Moreover, the app can function in any city that provides pollution data due to ODEN’s standardized data sets.”
ODEN’s origin
The idea for ODEN came in 2015 when D’Arcy and his company TerraTap were looking to enhance their product, neartuit, a location-based exhibit application, to include public art pieces from city Open Data.
“We realized we needed a community where individuals with common interests could collaborate.
D’Arcy: “We found that each city used a different format when gathering the data. We realized it would be a mammoth task cleaning up the data. Additionally, one of the cities kept updating their data and altering field names, continuously breaking the code. These experiences shed light on developers’ challenges in finding and utilizing city OD. It became clear that there was a need for a solution.”
“However, standardizing all the data is too large for a single person or a small group to handle alone. We realized we needed a community where individuals with common interests could come together and collaborate.”
ODEN’s quest & the help of Computing students
The following year, D’Arcy embarks on a long-term commitment and recruits the help of Computing’s Industry Sponsored Student Projects (ISSP) program, where senior Computing students work on real IT challenges given by industry. This time, however, D’Arcy is not the student’s supervisor but the client.
Initially, the project focuses on solidifying the idea of standardizing data. It culminates in a straightforward app that displays electric vehicle charging stations in British Columbia. The result is a promising glimpse of ODEN’s potential, which garners positive feedback from New Westminster and other cities with Smart City initiatives.
“The collaboration, adaptability, and teamwork exhibited were exceptional, showcasing these [students] future professionals’ potential.”
From that point on, D’Arcy conducts several more ISSP terms to further explore the idea of OD standardization.
Fast forward to the present.
Over the years, 40 CST students have put in nearly 3,000 hours altogether, including the last team which had no less than 19 students. And the work paid off because there is now a working prototype!
“Our Computing students have played a vital role in ODEN’s development,” says D’Arcy. Their time and skills have been crucial in turning our vision of standardized, easily accessible OD into a reality.”
“We allowed students considerable freedom to implement what they thought was best, encouraging true collaboration. They frequently surprised us with new ideas or took our thoughts in directions we hadn’t anticipated.”
ODEN’s future
A standout moment came from the final project, where 19 students worked on four separate but interconnected components. Each team had a leader who had started the project in a previous term, guiding the other members and mirroring a start-up environment.
D’Arcy: “The collaboration, adaptability, and teamwork exhibited during this phase were exceptional, showcasing these future professionals’ potential.”
Although there is a working prototype, a few challenges remain before ODEN can officially be launched, such as finalizing data privacy and copyright policies, creating a community interaction platform, and deciding on a file format for the Transmogrifier.
Setting a timeline for this ambitious project is challenging, says D’Arcy, but the next term in September with 15 students is already set.
“In an ideal world, ODEN will become the go-to hub for anyone seeking city OD,” says an excited D’Arcy. “The platform will nurture a dynamic community of volunteers dedicated to incorporating new cities and crafting schemas for datasets.”
“Cities won’t have to bear the burden of developing filters to adapt their data to match the schema; instead, the community will write filters or, better yet, cities can supply their datasets to comply with the schemas right out of the box. Conversely, developers will find it simpler to create innovative apps without having to focus on how to get usable data.”
Meet CST students Elaine Lai and Maxwell Babey
“I really enjoyed the ISSP term! When I heard we would be a group of 19, I thought it would be chaotic. However, we were organized into different teams (front-end, back-end, mobile, and Transmogrifier), and each group consisted of 3-5 members, each with a team lead. I think having that structure helped the organization of the project.
A highlight of working on ODEN was learning about D’Arcy’s approach to building a project. His approach is to create the prototype, knowing it will fail, and from there, identify and address the pain points during the second iteration.
I also really believed in ODEN’s overall goal of standardizing OD. I loved how we were able to be a part of this project and how I will get a chance to continue seeing it grow next semester.”
Elaine Lai
“Since I had experience working on this project in the winter term, I helped new students in the spring term get organized, divided them into teams and got them started on assignments. The team leads and I were able to put together a list of points where each of our teams would overlap – think of a big Venn diagram. Our planning prevented a lot of start-up issues.
I had to make a lot of moves very quickly to quell the wild chaos at the beginning of the term. I learned a lot about project management.
It was exhilarating.
In retrospect, I recognize a critical trade-off: the work needs to be done, so ruthlessness is needed; the team needs to grow and learn, so patience is required. Striking that balance is important.
For all future ISSP students: envision all the ways the project will fall apart and deal with it immediately. Coping with it requires communication with teammates, planning, and pivoting without complaint. Keep moving forward and be ready to compromise.
Interested in working with our Computing students?
The next ISSP term starts in September 2023, where Computing students will take on about 70 projects. If you have an IT project that needs a push to get it jumpstarted or finished, consider working with BCIT Computing students.
ISSP is a low-cost way to support your industry IT projects, and it’s free if you are a non-profit. Have a look at our past projects to get inspired and submit your project before August 1.