The challenge

Understanding and using data

Public data is a critically under-utilised national asset. The Australian Government's data holdings are a strategic national resource with considerable value for growing the economy - the better that the government can understand and use its data, the better it is able to make decisions and discover new opportunities. Many government agencies hold massive quantities of data, but it often gets stuck inside silos where its importance is invisible, origins untracked, and existence unknown to those elsewhere in the organisation who could improve or derive further value from it.

Outside of Government, there is considerable potential for public data to be openly published, and further utilised by the private sector, researchers and engaged citizens. The Australian Government is one of the world leaders in making government datasets open, but for the public to make use of the data, they need well-described and up-to-date data and powerful discovery tools in order to quickly find the data that is relevant to them.

Our response

Providing open-source software

Magda is a data catalog system that provides a single place where all of an organisation's data can be catalogued, enriched, searched, tracked and prioritised - whether big or small, internally or externally sourced, available as files, databases or APIs. 

Magda resolves the challenge of storing and using data by providing open-source software to assist government agencies in all aspects of data management—from collection and authoring, through internal discovery and use, all the way to sharing inter-organisation or publishing on an open data portal. In this manner it unlocks the potential of Australia's data, making it easier to find, access and reuse from both inside and outside the Australian Government.

Magda is supported under the Platforms for Open Data (PFoD) program which is a National Innovation and Science Agency (NISA) initiative. PFoD partners include the Digital Transformation Agency and the Department of Agriculture, Water and the Environment.

The results

Improving data use and sharing within the Australian Government

As part of discovery, the Magda team undertook extensive research in order to understand the current state of data sharing, pain points and future potential, conducting nearly 100 interviews with data users across the public, private and research sectors. The Magda team identified twelve opportunity areas where Magda could alleviate current state pain points and gaps around data sharing and access - this research has gone on to help inform the Office of the National Data Commissioner's efforts around improving data use and sharing within the Australian Government.

MAGDA Technology

MAGDA Technology

Since January 2019, Magda has been powering Australia's national data catalogue, data.gov.au .  Data.gov.au now makes over 89,000 datasets drawn from sources all over the Australian government available for search, and makes these available to 10,000-20,000 individual users per week.

As an open-source project, Magda has also been used by other teams within CSIRO, including for the Knowledge Network project in Land and Water, and the NSW Spatial Digital Twin developed by the Terria team within CSIRO's Data61.

Over the past 18 months, Magda has been extending the functionality initially developed for data.gov.au into a more general-purpose data cataloguing solution for use within government agencies. Working prototypes have been developed and deployed for the use of the Department of Agriculture, Water and Environment (DAWE), and will continue to be added to in an agile manner.  Of particular note is tackling the problem of making metadata around datasets easier and quicker to create for non-expert users.

The Magda team are looking forward to continuing to support data.gov.au ,  enhancing its reputation as an open-source project and rounding out its new metadata generation tool to DAWE, which is of particular use to the team writing the next State of the Environment report.

Do business with us to help your organisation thrive

We partner with small and large companies, government and industry in Australia and around the world.

Contact us now to start doing business

Contact Data61

Your contact details

First name must be filled in

We'll need to know what you want to contact us about so we can give you an answer.