hello,
I am executing a dataflow from a pipeline...
dataflow on its own which writes to delta parquet in azure storage works fine in debug when testing it on its own.,
The dataflow, simply joins .csv files together and write to delta...
When I execute it from a pipeline, I see this message and not sure what is causing this error.
Any thoughts please? thank you

Error details
Error code
DFExecutorUserError
Failure type
User configuration issue
Details
Job failed due to reason: None.get
Source
Pipeline
pipeline1
Data flow
df_testflow
Monitor
Data flow activity
Data flow1

Hi again @arkiboys . This one looks weird. So as I understand the dataflow works fine in a debug run, but a triggered pipeline fails with that error?

Hmm, well the difference between a debug's run and a trigger's run, is where the instructions are fetched from. Can you try doing a publish? Maybe the live version isn't the same as the displayed version.
If it had never been published, I would expect a different error message. None.get means something important is missing. Does this error happen every time or just once?

If this was just a mistake in the data flow script or data I would expect a longer error message. Something so short smells like a problem closer to the runtime.

I have the exact same problem. The dataflow works flawlessly in debug mode, wether it's live mode or not. Only stops working when the pipeline is run and only with that dataflow. Same error message.

"StatusCode": "DFExecutorUserError",
"Message": "Job failed due to reason: None.get",
"Details": "java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at com.microsoft.datafactory.dataflow.MetricsUtility$$anonfun$getMetricsForSink$5.apply(MetricsUtility.scala:306)
at com.microsoft.datafactory.dataflow.MetricsUtility$$anonfun$getMetricsForSink$5.apply(MetricsUtility.scala:295)
at scala.collection.immutable.List.foreach(List.scala:392)
at com.microsoft.datafactory.dataflow.MetricsUtility$.getMetricsForSink(MetricsUtility.scala:295)
at com.microsoft.datafactory.dataflow.MetricsUtility$.metricsToPayload(MetricsUtility.scala:463)
at com.microsoft.datafactory.dataflow.AdmsClient.getMonitoringPayloadInternal(AdmsClient.scala:485)
at com.microsoft.datafactory.dataflow.AdmsEventListener$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$8$$anonfun$apply$1.apply(AdmsEventListener.scala:90)
at com.microsoft.datafactory.dataflow.AdmsEventListener$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$8$$anonfun$apply$1.apply(AdmsEventListener.scala:90)
at sca"