data.all

What it does

Administration

Administrators can organize and manage teams and users across environments.checked 8 Oct 2001

Analytics and ML

The platform provides Athena worksheets, Jupyter notebooks through SageMaker, SageMaker Studio access, and QuickSight dashboards.checked 8 Oct 2001

Audit

For structured data, SQL queries from EMR, Redshift, Glue Jobs, and Athena are captured through Lake Formation.checked 8 Oct 2001

AWS integrations

The documented architecture uses AWS services including Cognito, CloudFront or an internal Application Load Balancer, API Gateway, Lambda, Aurora PostgreSQL, ECS Fargate, and OpenSearch.checked 8 Oct 2001

Catalog

Its centralized catalog inventories datasets, tables, folders, and dashboards, with search filters for asset type, tags, region, and glossary terms.checked 8 Oct 2001

Configuration

Modules and some features can be enabled or disabled through configuration, and the guide notes that file uploads are disabled in its example configuration.checked 8 Oct 2001

Data catalog

Its centralized catalog inventories datasets, tables, folders, and dashboards, with search filters for asset type, tags, region, and glossary terms.checked 8 Oct 2001

Datasets

Datasets can hold structured tables or unstructured folders and use AWS S3, KMS, IAM, and Glue resources.checked 8 Oct 2001

Data sharing

Dataset sharing across environments can grant selected team roles read access through Lake Formation without moving the data.checked 8 Oct 2001

Deployment

The AWS deployment guide requires a tooling account and at least one deployment account, and their selected regions must match.checked 8 Oct 2001

Deployment options

The deployment guide describes internet-facing and VPC-facing architectures; the VPC-facing option makes the UI and APIs reachable only within the VPC.checked 8 Oct 2001

Identity

Authentication can use Amazon Cognito federated with a corporate identity provider, or an external OIDC provider; data.all relies on identity-provider group information.checked 8 Oct 2001

Integrations

Documented modules include AWS Glue, Lake Formation, Amazon S3, Athena, SageMaker, QuickSight, and HealthOmics Ready2Run workflows.checked 8 Oct 2001

Local use

The project documents a Docker Compose local setup with frontend, database, GraphQL, CDK proxy, and OpenSearch containers; AWS credentials are needed to create environments and datasets.checked 8 Oct 2001

Network options

Deployments can be internet facing, with public APIs protected by AWS WAF, or VPC facing, with private sites and APIs accessible through VPC endpoints.checked 8 Oct 2001

Notebooks

The platform provides notebooks based on Amazon SageMaker Studio for data scientists to build machine learning models.checked 8 Oct 2001

Operational limit

The AWS deployment instructions require administrator credentials for the tooling and deployment accounts during setup.checked 8 Oct 2001

Purpose

data.all is an open source platform that can be deployed in AWS accounts to help users manage and discover data assets.checked 8 Oct 2001

Redshift integration

It supports importing tables from Redshift Serverless and provisioned clusters into the catalog; connections can use Secrets Manager secrets or Redshift users.checked 8 Oct 2001

Requirements

AWS deployment requires at least two AWS accounts and administrator credentials for the tooling and deployment accounts.checked 8 Oct 2001

Security

Dataset S3 buckets are encrypted with AWS managed KMS keys, and data access is logged through CloudTrail.checked 8 Oct 2001