I'm experimenting with TFX for common ML pipeline work. I somewhat struggle to actually utilize StatisticsGen component to inspect an analyze data statistics.
While in case of TFDV I can access statistics in a straightforward manner:
import tensorflow_data_validation as tfdv
stats = tfdv.generate_statistics_from_csv('data.csv', delimiter=',')
stats # This gives a JSON-like output
in case of TFX itself, StatisticsGen generates a binary FeatureStats.pb file in artifacts/StatisticsGen/statistics/...
How to extract actual statistics from StatisticsGen to use it for checking data (or any other purpose)? I'm aware of the existence of interactive context's ability to visualize stats, but this is unhelpful in production environment.
Related
I’d like to be able to maintain a grouping of entities within a single PCollection element, but parallelize the fetching of those entities from Google Cloud Storage (GCS). i.e.PCollection<Iterable<String>> --> PCollection<Iterable<String>> where the starting PCollection is an Iterable of file paths and the resulting PCollection is Iterable of file contents. Alternatively, PCollection<String> --> PCollection<Iterable<String>> would also work and perhaps even be preferable, where the starting PCollection is a glob pattern, and the resulting PCollection is an iterable of file contents which matched the glob.
My use-case is that at a point in my pipeline I have as input PCollection<String>. Each element of the PCollection is a GCS glob pattern. It’s important that files which match the glob be grouped together because the content of the files–once all files in a group are read–need to be grouped downstream in the pipeline. I originally tried using FileIO.matchAll and a subsequently GroupByKey . However, the matchAll, window, and GroupByKey combination lacked any guarantee that all files matching the glob would be read and in the same window before performing the GroupByKey transform (though I may be misunderstanding Windowing). It’s possible to achieve the desired results if a large time span WindowFn is applied, but it’s still probabilistic rather than a guarantee that all files will be read before grouping. It’s also the main goal of my pipeline to maintain the lowest possible latency.
So my next, and currently operational, plan was to use an AsyncHttpClient to fan out fetching file contents via GCS HTTP API. I feel like this goes against the grain in Beam and is likely sub-optimal in terms of parallelization.
So I’ve started investigating SplittableDoFn . My current plan is to allow splitting such that each entity in the input Iterable (i.e. each matched file from the glob pattern) could be processed separately. I've been able to modify FileIO#MatchFn (defined here in the Java SDK) to provide mechanics for PCollection<String> -> PCollection<Iterable<String>> transform between input of GCS glob patterns and output of Iterable of matches for the glob.
The challenge I’ve encountered is: how do I go about grouping/gathering the split invocations back into a single output value in my DoFn? I’ve tried using stateful processing and using a BagState to collect file contents along the way, but I realized part way along that the ProcessElement method of a splittable DoFn may only accept ProcessContext and Restriction tuples, and no other args therefore no StateId args referring to a StateSpec (throws an invalid argument error at runtime).
I noticed in the FilePatternWatcher example in the official SDF proposal doc that a custom tracker was created wherein FilePath Objects kept in a set and presumably added to the set via tryClaim. This seems as though it could work for my use-case, but I don’t see/understand how to go about implementing a #SplitRestriction method using a custom RestrictionTracker.
I would be very appreciative if anyone were able to offer advice. I have no preference for any particular solution, only that I want to achieve the ability to maintain a grouping of entities within a single PCollection element, but parallelize the fetching of those entities from Google Cloud Storage (GCS).
Would joining the output PCollections help you?
PCollectionList
.of(collectionOne)
.and(collectionTwo)
.and(collectionThree)
...
.apply(Flatten.pCollections())
I'm getting an error whenever I use "WriteToPubSub". The code below is me trying to debug the issue. My actual code is trying to take data from failures of WriteToBigQuery in order to push it to a deadletter pubsub topic. But when I tried to do that I kept encountering the error below.
I am running Apache Beam 2.27, Python 3.8
import apache_beam as beam
from apache_beam.runners.interactive.interactive_runner import InteractiveRunner
from apache_beam.io.gcp.bigtableio import WriteToBigTable
from apache_beam.runners import DataflowRunner
import apache_beam.runners.interactive.interactive_beam as ib
from apache_beam.options import pipeline_options
from apache_beam.options.pipeline_options import GoogleCloudOptions
import google.auth
import json
import pytz
# Setting up the Apache Beam pipeline options.
options = pipeline_options.PipelineOptions(flags=[])
# Sets the project to the default project in your current Google Cloud environment.
_, options.view_as(GoogleCloudOptions).project = google.auth.default()
# Sets the Google Cloud Region in which Cloud Dataflow runs.
options.view_as(GoogleCloudOptions).region = 'asia-east1'
# Sets the job name
options.view_as(GoogleCloudOptions).job_name = 'data_ingest'
# IMPORTANT! Adjust the following to choose a Cloud Storage location.
dataflow_gcs_location = '[REDACTED]'
# Dataflow Staging Location. This location is used to stage the Dataflow Pipeline and SDK binary.
options.view_as(GoogleCloudOptions).staging_location = '%s/staging' % dataflow_gcs_location
# Dataflow Temp Location. This location is used to store temporary files or intermediate results before finally outputting to the sink.
options.view_as(GoogleCloudOptions).temp_location = '%s/temp' % dataflow_gcs_location
# The directory to store the output files of the job.
output_gcs_location = '%s/output' % dataflow_gcs_location
ib.options.recording_duration = '1m'
# The Google Cloud PubSub topic for this example.
topic = "[REDACTED]"
output_topic = "[REDACTED]"
subscription = "[REDACTED]"
deadletter_topic = "[REDACTED]"
class PrintValue(beam.DoFn):
def process(self, element):
print(element)
return [element]
p = beam.Pipeline(InteractiveRunner(),options=options)
data = p | beam.io.ReadFromPubSub(topic=topic) | beam.ParDo(PrintValue()) | beam.io.WriteToPubSub(topic=deadletter_topic)
ib.show(data, include_window_info=False)
The error given is
ValueError: The given pcoll PDone[WriteToPubSub/Write/NativeWrite.None] is not a dict, an iterable or a PCollection.
Can someone spot what the problem is?
No matter what I do, WriteToPubSub says it's receiving PDone.
EDIT:
If i use p.run(), I get the following error instead:
'PDone' object has no attribute 'to_runner_api'
In both cases, the pipeline does not try to run, it immediately errors out.
EDIT:
I've realised the problem
p = beam.Pipeline(InteractiveRunner(),options=options)
It is this line. If I remove the interactiverunner everything works. Not sure why
Beam Terminology
Apache Beam has some base concepts, that we should adhere to while leveraging the power of this programming model.
Pipeline
In simple terms, a pipeline is a series of tasks for a desired output. It can be as simple as a linear flow or could have a complex branching of tasks. The fundamental concept is read from input source(s), perform some transformations and emit to output(s).
Mathematically, beam pipeline is just a Directed Acyclic Graph of tasks.
PCollection
In simple terms, PCollections is an immutable bag of elements which could be distributed across machines. Each step in a beam pipeline will have it's input and output as a PCollection (apart from sources and sinks)
PCollection is a powerful distributed data structure that a beam pipeline operates on. It could be bounded or unbounded based on your source type.
PTransforms
In simple terms, Transforms are the operations of your pipleine. It provides processing logic and this logic is applied to each element of one or more input of PCollections.
Example : PTransform<PCollection<X>,PCollection<Y>> will transform X to Y.
Based on processing paradigm, beam provides us multiple core transforms - ParDo, GroupByKey, Flatten, Combine etc.
I/O Transforms
When you create a pipeline one need a data source to read data such as a file or a database. Likewise, you want to emit your result data to an external storage system such as topic or an object store. The transforms which deal with External Input and Output are I/O Transforms.
Usually for an external source, you will have the following
Source : A PTransform to read data from the external system. This will read from
an external system(like file, db). It excepts a PBegin (pipeline entry point) and return a PCollection.
PTransform<PBegin,PCollection>
This would be one of the entry points of your pipeline.
Sink : A PTransform that will output data to an external system. This will write to an external system(like topic, storage). It excepts a PCollection and return a PDone (pipeline entry point).
PTransform<PCollection,PDone>
This would be one of the exit points of your pipeline.
Combination of a source and sink is an I/O Connector like RedisIO, PubSubIO etc. Beam provides multiple in-built connectors and one can write a custom one also.
There are still various concepts and extenions of the above, that allow users to program complex requirements that could be run on different runners. This is what makes Beam so powerful.
Solution
In your case, ib.show(data, include_window_info=False) is throwing the below error
ValueError: The given pcoll PDone[WriteToPubSub/Write/NativeWrite.None] is not a dict, an iterable or a PCollection.
Source Code
Because your data contains result of beam.io.WriteToPubSub(topic=deadletter_topic) which is a sink and returns a PDone not a PCollection.
For your use case of BQ Writing Failures to PubSub, you could follow something below
data = beam.io.ReadFromPubSub(topic=topic) | 'Write to BQ' >> beam.io.WriteToBigQuery( ...)
(data['beam.io.gcp.bigquery.BigQueryWriteFn.FAILED_ROWS']
| 'publish failed' >> beam.io.WriteToPubSub(topic=deadletter_topic)
However, if this does not solve your issue posting the code would be useful or else you could write a custom PTransform with output tags for writing to BQ and to return failures(via tuple tags) for publising to PubSub.
P.S. : WriteToBigQuery is not a sink, but a custom PTransform that writes to big query and returns failures.
I am very impressed with Dask and I am trying to determine if it is the right tool for my problem. I am building a project for interactive data exploration where users can interactively change parameters of a figure. Sometimes these changes requires re-computing the entire pipeline to make the graph (e.g. "show data from a different time interval"), but sometimes not. For instance, "change the smoothing parameter" should not require the system to reload the raw unsmoothed data, because the underlying data is the same, only the processing changes. The system should instead use the existing raw data that has already been loaded. I would like my system to be able to keep around the intermediate data objects and intelligently determine what tasks in the graph need to be re-run based on what parameters of the data visualization have been changed. It looks like the caching system in Dask is close to what I need, but was designed with a bit of a different use-case in mind. I see there is a persist method, but I'm not sure if that would work either. Is there an easy way to accomplish this in Dask, or is there another project that would be more appropriate?
"change the smoothing parameter" should not require the system to reload the raw unsmoothed data
Two options:
The builtin functools.lru_cache will cache every unique input. The check on memory is with the maxsize parameter, which controls how many input/output pairs are stored.
Using persist in the right places will compute that object as mentioned at https://distributed.dask.org/en/latest/manage-computation.html#client-persist. It will not require re-running computation to get the object in later computation; functionally, it's the same as lru_cache.
For example, this code will read from disk twice:
>>> import dask.dataframe as dd
>>> df = dd.read_csv(...)
>>> # df = df.persist() # uncommenting this line → only read from disk once
>>> df[df.x > 0].mean().compute()
24.9
>>> df[df.y > 0].mean().compute()
0.1
Uncommented the line will mean this code only reads from disk once because the task graph for the CSV is computed and the value is stored in memory. For your application is sounds like I would use persist intelligently: https://docs.dask.org/en/latest/best-practices.html#persist-when-you-can
What if two smoothing parameters want to be visualized? In that case, I'd avoid calling compute repeatedly: https://docs.dask.org/en/latest/best-practices.html#avoid-calling-compute-repeatedly
lower, upper = client.compute(df.x.min(), df.x.max())
This will share the task graph for min and max so unnecessary computation is not performed.
I would like my system to be able to keep around the intermediate data objects and intelligently determine what tasks in the graph need to be re-run based on what parameters of the data visualization have been changed.
Dask Distributed has a smart caching ability: https://docs.dask.org/en/latest/caching.html#automatic-opportunistic-caching. Part of the documentation says
Another approach is to watch all intermediate computations, and guess which ones might be valuable to keep for the future. Dask has an opportunistic caching mechanism that stores intermediate tasks that show the following characteristics:
Expensive to compute
Cheap to store
Frequently used
I think this is what you're looking for; it'll store values depending on those attributes.
This topic is difficult to Google, because of "node" (not node.js), and "graph" (no, I'm not trying to make charts).
Despite being a pretty well rounded and experienced developer, I can't piece together a mental model of how these sorts of editors get data in a sensible way, in a sensible order, from node to node. Especially in the Alteryx example, because a Sort module, for example, needs its entire upstream dataset before proceeding. And some nodes can send a single output to multiple downstream consumers.
I was able to understand trees and what not in my old data structures course back in the day, and successfully understand and adapt the basic graph concepts from https://www.python.org/doc/essays/graphs/ in a real project. But that was a static structure and data weren't being passed from node to node.
Where should I be starting and/or what concept am I missing that I could use implement something like this? Something to let users chain together some boxes to slice and dice text files or data records with some basic operations like sort and join? I'm using C#, but the answer ought to be language independent.
This paradigm is called Dataflow Programming, it works with stream of data which is passed from instruction to instruction to be processed.
Dataflow programs can be programmed in textual or visual form, and besides the software you have mentioned there are a lot of programs that include some sort of dataflow language.
To create your own dataflow language you have to:
Create program modules or objects that represent your processing nodes realizing different sort of data processing. Processing nodes usually have one or multiple data inputs and one or multiple data output and implement some data processing algorithm inside them. Nodes also may have control inputs that control how given node process data. A typical dataflow algorithm calculates output data sample from one or many input data stream values as for example FIR filters do. However processing algorithm also can have data values feedback (output values in some way are mixed with input values) as in IIR filters, or accumulate values in some way to calculate output value
Create standard API for passing data between processing nodes. It can be different for different kinds of data and controlling signals, but it must be standard because processing nodes should 'understand' each other. Data usually is passed as plain values. Controlling signals can be plain values, events, or more advanced controlling language - depending of your needs.
Create arrangement to link your nodes and to pass data between them. You can create your own program machinery or use some standard things like pipes, message queues, etc. For example this functional can be implemented as a tree-like structure whose nodes are your processing nodes, and have references to next nodes and its appropriate input that process data coming from the output of the current node.
Create some kind of nodes iterator that starts from begin of the dataflow graph and iterates over each processing node where it:
provides next data input values
invokes node data processing methods
updates data output value
pass updated data output values to inputs of downstream processing nodes
Create a tool for configuring nodes parameters and links between them. It can be just a simple text file edited with text editor or a sophisticated visual editor with GUI to draw dataflow graph.
Regarding your note about Sort module in Alteryx - perhaps data values are just accumulated inside this module and then sorted.
here you can find even more detailed description of Dataflow programming languages.
I have built an Microsoft Azure ML Studio workspace predictive web service, and have a scernario where I need to be able to run the service with different training datasets.
I know I can setup multiple web services via Azure ML, each with a different training set attached, but I am trying to find a way to do it all within the same workspace and passing a Web Input Parameter as the input value to choose which training set to use.
I have found this article, which describes almost my scenario. However, this article relies on the training dataset that is being pulled from the Load Trained Data module, as having a static endpoint (or blob storage location). I don't see any way to dynamically (or conditionally) change this location based on a Web Input Parameter.
Basically, does Azure ML support a "conditional training data" loading?
Or, might there be a way to combine training datasets, then filter based on the passed Web Input Parameter?
This probably isn't exactly what you need, but hopefully, it helps you out.
To combine data sets, you can use the Join Data module.
To filter, that may be accomplished by executing a Python script. Here's an example.
Using the Adult Census Income Binary Classification dataset, on the age column, there's a minimum age of 17.
If I wanted to filter the data set by age, connect it to an Execute Python Script module and here's the filtering code with the pandas query method.
# The script MUST contain a function named azureml_main
# which is the entry point for this module.
import pandas as pd
def azureml_main(dataframe1 = None, dataframe2 = None):
# Return value must be of a sequence of pandas.DataFrame
return dataframe1.query("age >= 25")
And looking at that output it filters out the data set where the minimum age is now 25.
Sure, you can do that. What you would want is to use an Execute R Script or SQL Transformation module to determine, based on your input data, what model to use. Something like this:
Notice, your input data is cleaned/updated/feature engineered, then it's passed to two different SQL transforms which will tell it to go to one of two paths.
Each path has it's own training data.
Note: I am not exactly sure what your use case is, but if it were me, I would instead train two different models using the two different training data, then try to just use the models in my web service, not actually train on the web service as that would likely be quite slow.