yikuan8/MIMIC-IV-Postgres

README

Installing MIMIC-IV-ED and MIMIC-IV in a local Postgres database

Jun 28th, 2021: The repo updates to MIMIC-IV V4.0. As there is a major update of MIMIC-IV from version 0.4 to 1.0, I modify some codes to accomodate these changes.

Jun 2nd, 2021: MIMIC-IV-ED is released. Shout out to the authors. If you don't need the ED data, just skip the steps numbered with "ED". If you have already installed MIMIC-IV using this repo, you only need to run 1.ED, 4.ED, 5.ED.

In light of there is no offcial guide of deploying MIMIC-IV to Postgres server, this repo will help you navigate this process. We made necessary modification based on the offcial guide of MIMIC-III to adapt to MIMIC-IV.

1. Prepare the data

Complete the PhysioNet Credentialing and download MIMIC-IV data. Decompress all .csv.gz file using "gzip -d *.gz". You will have such directory structure:

BASE_DIR  
│
└───core
│   │   admissions.csv
│   │   patients.csv
│   │   transfers.csv
└───hosp
│   │   d_hcpcs.csv
│   │   ...
└───icu
│   │   d_items.csv
│   │   ...

1.ED Prepare the ED data

Complete the PhysioNet Credentialing and download [MIMIC-IV-ED data][MIMIC-IV_ED]. Decompress all .csv.gz file using "gzip -d *.gz". Move those decompressed file to the same base directory. You will have such directory structure:

BASE_DIR  
│
└───core
│   │   admissions.csv
│   │   patients.csv
│   │   transfers.csv
└───hosp
│   │   d_hcpcs.csv
│   │   ...
└───icu
│   │   d_items.csv
│   │   ...
└───ed
│   │   edstays.csv
│   │   ...

2. Install Postgresql server

Please refer here for postgresql download and installation

3. Create user, database, schema

In my case, I create a user named mimicuser, a new database named mimic4 containing a new schema named mimiciv holding all data by:

createuser -P -s -e -d mimicuser             #creat user
psql -U mimicuser                            #launch psql
CREATE DATABASE mimic4 OWNER mimicuser;      #create new database
\c mimic4;                                   #enter the new database
CREATE SCHEMA mimiciv;                       #create new schema
\q                                           #quit psql

4. Creat a set of empty tables with create_tables.sql

psql 'dbname=mimic4 user=mimicuser options=--search_path=mimiciv' -f create_tables.sql

4.ED Creat a set of empty ED tables with create_ed_tables.sql

psql 'dbname=mimic4 user=mimicuser options=--search_path=mimiciv' -f create_ed_tables.sql

5. load .csv files into the empty tables with load_data.sql

Change /YOUR/BASE/DIR/ to your BASE_DIR in the line 3,11,27 and run:

psql 'dbname=mimic4 user=mimicuser options=--search_path=mimiciv' -f load_data.sql

5.ED load .csv files into the empty ED tables with load_ed_data.sql

Change /YOUR/BASE/DIR/ to your BASE_DIR in the line 3 and run:

psql 'dbname=mimic4 user=mimicuser options=--search_path=mimiciv' -f load_eddata.sql

NOTES:

The data description and real data are inconsistent in data format, varchar length, etc. This repo is implemented based on the real data. Some comments are left in create_tables.sql to mark down the difference.
We follow the installation of MIMIC-III that creating 10 subtables for labevents.csv and chartevents.csv, respectively. Chartevents table is divided by itemid to 10-fold, while labevents table is divided by labevents_id.
The size of MIMIC-IV after de-compression may exceed 100GB. Therefore, loading data is time-consuming. You may expect several hours to run through the process.
This repo is based on the latest data release before Jun 28, 2021. Should there be any data update, incompatibiltiy may observed.
Leave an issue, if you encounter any error.