Skip to content

Real Time Data Ingestion Platform

core

Initializing search

rtdip/core

Home
Getting started
SDK
API
Domains
Integration
Roadmaps
Releases
Blog
University

Real Time Data Ingestion Platform

rtdip/core

Home
Getting started
Getting started
- About RTDIP
- Installation
SDK
SDK
- Overview
- Authentication
  Authentication
  - Azure
    Azure
    
    Overview
    
    Azure Active Directory
  - Databricks
- Pipelines
  Pipelines
  - Overview
  - Components
    Components
    
    Overview
    
    Sources
    Sources
    
    Spark
    Spark
    
    Eventhub
    
    Eventhub Kafka
    
    Delta
    
    Delta Sharing
    
    Autoloader
    
    IoT Hub
    
    Kafka
    
    Kinesis
    
    ISO
    ISO
    
    MISO Daily Load
    
    MISO Historical Load
    
    PJM Daily Load
    
    PJM Historical Load
    
    CAISO Daily Load
    
    CAISO Historical Load
    
    ERCOT Daily Load
    
    PJM Daily Pricing Load
    
    PJM Historical Pricing Load
    
    Weather
    Weather
    
    Base Weather
    
    Weather Forecast API V1
    
    Weather Forecast V1 Multi
    
    ECMWF
    ECMWF
    
    Base MARS
    
    Weather Forecast
    
    Python
    Python
    
    Delta
    
    Delta Sharing
    
    ENTSO-E
    
    MFFBAS
    
    Transformers
    Transformers
    
    Spark
    Spark
    
    Binary To String
    
    OPC Publisher Json To Process Control Data Model
    
    OPC Publisher Json for A&E(Alarm & Events) Data to Process Control Data Model
    
    Fledge Json To Process Control Data Model
    
    EdgeX JSON data To Process Control Data Model
    
    SEM data To Process Control Data Model
    
    SSIP PI Binary File data To Process Control Data Model
    
    SSIP PI Binary JSON data To Process Control Data Model
    
    Pandas to PySpark DataFrame Conversion
    
    PySpark to Pandas DataFrame Conversion
    
    Honeywell APM To Process Control Data Model
    
    Process Control Data Model To Honeywell APM
    
    Mirico data To Process Control Data Model
    
    ISO
    ISO
    
    MISO To Meters Data Model
    
    PJM To Meters Data Model
    
    ERCOT To Meters Data Model
    
    CAISO To Meters Data Model
    
    The Weather Company
    The Weather Company
    
    Raw Forecast To Weather Data Model
    
    ECMWF
    ECMWF
    
    NC Forecast Extract Base To Weather Data Model
    
    NC Forecast Extract Point To Weather Data Model
    
    NC Forecast Extract Grid To Weather Data Model
    
    Destinations
    Destinations
    
    Spark
    Spark
    
    Delta
    
    Delta Merge
    
    Eventhub
    
    Eventhub Kafka
    
    Kafka
    
    Kinesis
    
    Rest API
    
    PCDM To Delta
    
    PCDM Latest To Delta
    
    Python
    Python
    
    Delta
    
    Blockchain
    Blockchain
    
    EVM
    
    Utilities
    Utilities
    
    Spark
    Spark
    
    Session
    
    Configuration
    
    Delta Table Create
    
    Delta Table Optimize
    
    Delta Table Vacuum
    
    ADLS Gen 2 Service Principal Connect
    
    AWS
    AWS
    
    S3 Bucket Policy
    
    S3 Copy Utility
    
    Azure
    Azure
    
    ADLS Gen 2 ACLs
    
    Autoloader Resources
    
    Converters
    Converters
    
    Json
    
    Secrets
    Secrets
    
    Databricks
    
    Hashicorp Vault
    
    Azure Key Vault
    
    Deploy
    Deploy
    
    Databricks
    
    Data Quality
    Data Quality
    
    Monitoring
    Monitoring
    
    Check Value Ranges
    
    Great Expectations
    Great Expectations
    
    Data Quality Monitoring
    
    Flatline Detection
    
    Identify Missing Data
    Identify Missing Data
    
    Interval Based
    
    Pattern Based
    
    Data Manipulation
    Data Manipulation
    
    Duplicate Detection
    
    Filter Out of Range Values
    
    Flatline Filter
    
    Dimensionality Reduction
    
    Interval Filtering
    
    K-Sigma Anomaly Detection
    
    Missing Value Imputation
    
    Normalization
    Normalization
    
    Normalization
    
    Normalization Mean
    
    Normalization MinMax
    
    Normalization ZScore
    
    Prediction
    Prediction
    
    Arima
    
    Auto Arima
    
    Machine Learning
    Machine Learning
    
    Data Binning
    
    Linear Regression
  - Jobs
  - Deploy
    Deploy
    
    Databricks Workflows
    
    Apache Airflow
  - Examples
    Examples
    
    Python Delta to Delta
    
    EdgeX Eventhub to Delta
    
    ECMWF to Delta
    
    Dagster
    Dagster
    
    Databricks Connect
    
    Local
    
    Databricks
    Databricks
    
    MISO Daily Load Databricks
    
    Spark Connect
    
    MISO Daily Load Local
    
    PJM Daily Load Local
- Queries
  Queries
  - Query Types
    Query Types
    
    Overview
    
    Metadata
    
    Time Series
    Time Series
    
    Time Series Query Builder
    
    Raw
    
    Latest
    
    Resample
    
    Plot
    
    Interpolate
    
    Interpolation at Time
    
    Time Weighted Average
    
    Circular Average
    
    Circular Standard Deviation
    
    Summary
    
    Weather
    Weather
    
    Weather Data Query Builder
    
    Weather Raw
    
    Weather Latest
  - SQL
  - Connectors
    Connectors
    
    Overview
    
    Databricks SQL Connector
    
    PYODBC SQL Connector
    
    TURBODBC SQL Connector
    
    Spark Connector
    
    Chat Open AI Databricks Connector
  - Databricks
    Databricks
    
    Databricks SQL
    
    SQL Warehouses
    
    Troubleshooting
  - Spark Connect
  - LLMs
    LLMs
    
    Chat Open AI Databricks
  - Examples
    Examples
    
    Interpolate
    
    Interpolation at Time
    
    Metadata
    
    Raw
    
    Resample
    
    Plot
    
    Time Weighted Average
    
    Circular Average
    
    Circular Standard Deviation
    
    Summary
API
API
- Overview
- Authentication
- Examples
- REST API Documentation
- Deployment
  Deployment
  - Azure
Domains
Domains
- Process Control
  Process Control
  - Overview
  - Data Model
- Smart Meter Data
  Smart Meter Data
  - Overview
  - Data Model
- Weather Data
  Weather Data
  - Overview
  - Data Model
Integration
Integration
- Power BI
Roadmaps
Roadmaps
- Overview
- 2023
- 2022
Releases
Releases
- core
Blog
Blog
- Archive
  Archive
  - 2024
  - 2023
  - 2022
University
University
- University

Releases