Database Administrators Stack Exchange
Database Administrators Stack Exchange

Questions tagged [databricks]

9 questions
2
votes
0 answers

Troubleshoting slow running queries/jobs in Azure Databricks

I have Azure Databricks workspace with cluster configured to run Standard 6.4 runtime (Apache Spark 2.4.5, Scala 2.11). Cluster uses shared metastore (Azure MySQL). I'm trying to figure out possible way to troubleshoot sporadically slow execution of…
Mike
  • 747
  • 8
  • 14
  • 25
1
vote
0 answers

Writing large dataset from spark dataframe

We have a azure databricks job that retrieves some large dataset with pyspark. The dataframe has about 11 billion rows. We are currently writing this out to a postgresql DB (also in azure). Currently we are using the jdbc connector to write row out…
Kyle Chamberlin
  • 13
  • 2
1
vote
1 answer

DESCRIBE TABLE in databricks piped into dataframe

Does anyone know of a method to pipe the "DESCRIBE TABLE" output in databricks into dataframe? (or other usable format which could be used for further analysis/computation)?
Doc
  • 121
  • 2
1
vote
1 answer

How to call python file in repo in databricks from data factory outside DBFS?

In Azure Databricks I have I have a repo cloned which contains python files, not notebooks. In Azure Data Factory I want to configure a step to run a Databricks Python file. However when I enter the /Repos/..../myfile.py (which works for…
Brendan Hill
  • 301
  • 1
  • 4
  • 11
0
votes
1 answer

How to count the number of campaigns per day based on the start and end dates of the campaigns

I need to count the number of campaigns per day based on the start and end dates of the campaigns Columns: Campaign Name, Start Date, End Date How do I need to write the SQL command in databricks?
Level11Data
  • 11
  • 1
0
votes
1 answer

Databricks SQL warehouse is failing to launch saying it "cannot fetch secrets", what is going on?

I have a Databricks SQL warehouse. When I try to start it, I get the following error: Clusters are failing to launch. Cluster launch will be retried. Details for the latest failure: Error: Cannot fetch secrets referred in the Spark configuration.…
Kyle Hale
  • 216
  • 1
  • 4
0
votes
0 answers

Connect to Create a New Unity Catalog using a onprem postgres database connect

Have datbricks on azure platform with admin acces. I have serverless sql warehouse where i have imported some csv data into a catalog. Now i need to access postgres data on onprem linux box. Need to connect this db from datbticks add connection to…
malcolm richard
  • 1
  • 1
0
votes
1 answer

Next Business Date Column

I have a dataset that looks like this. Where business_day indicates whether the transaction_created_date is a business day or not. I'm trying to sum the line_amount so that values that occurred over the holiday or weekend gets added to the next…
Lena Zheng
  • 3
  • 1
-1
votes
1 answer

How to create "On this day in history" query

I'm using Databricks and I have a table with a list of event from various years. I want to return the event most recent to today's date from each year. For example, Today's date is 6th May and my table is thus: Year (int) Date (date) Event…
ben_al
  • 1
  • 1