Database Administrators Stack Exchange
Database Administrators Stack Exchange

Questions tagged [polybase]

Microsoft SQL Server product to allow SQL queries against relational, Hadoop and Azure blob storage.

PolyBase is a technology that accesses and combines both non-relational and relational data, all from within SQL Server. It allows you to run queries on external data in Hadoop or Azure blob storage. The queries are optimized to push computation to Hadoop.

26 questions
7
votes
1 answer

Performance of Polybase

We've been experimenting with Polybase on a SQL Server 2019 (CU2) box, using SQL Server external data sources and the performance has not been good - in most cases a 1400% increase. In each case all of the tables/views we're querying are from…
Select 'DBA'
  • 165
  • 2
  • 11
5
votes
3 answers

Azure SQL Warehouse - Data Ingestion - Convert a huge fixed width (with commas) file to delimited

I am not even sure if I am framing this question right, but I will try - I have a bunch of huge text files generated from an Oracle export on a Linux system. Each file is about 30 GB in size, and I have about 50 of them. The goal is to export this…
RK Kuppala
  • 2,447
  • 1
  • 22
  • 24
4
votes
4 answers

Why is Microsoft SQL Server creating a ton of dump files?

How to determine what is causing Microsoft SQL Server to create dump files over and over taking up 170 GB of disk space? How do I troubleshoot what is causing the dumps? The path of the dump files is C:\Program Files\Microsoft SQL…
Tanooki Mario
  • 41
  • 1
  • 1
  • 3
3
votes
1 answer

Can't stop "SQL Server Polybase Data Movement"

When I start Windows the "SQL Server Polybase Data Movement" service is in the "Change pending..." state and I can't close it. When I try to stop it mmc crashes. In Windows services, the status = starting, but it never changes either. I cannot stop…
xenorack
  • 33
  • 1
  • 5
3
votes
1 answer

How does PolyBase collect statistics from external tables?

Books online has this to say about query optimisation in PolyBase: The query optimizer makes a cost-based decision to push computation to Hadoop when doing so will improve query performance. It uses statistics on external tables to make the…
Michael Green
  • 25,255
  • 13
  • 54
  • 100
2
votes
0 answers

Does polybase support having a table to represent a REST API?

I have customer information in a SQL server table, and the billing information is exposed via REST web api (with customer_id as parameter). I am exploring using polybase to create the polybase table to represent the REST api, so that I can perform a…
variable
  • 3,590
  • 4
  • 37
  • 100
2
votes
1 answer

PolyBase :: How to setup EXTERNAL DATA SOURCE through ODBC?

Microsoft states that: [...You can also use the generic ODBC connector to connect to additional providers using third-party ODBC drivers.] The reality is much far away form that. I've been trying to follow several guides about how to query DB2…
Francesco Mantovani
  • 1,695
  • 14
  • 28
2
votes
3 answers

SQL Server Polybase Data Movement Service stopping after few seconds

I have been scratching my head for a while now on how to fix this issue, SQL Server Polybase Data Movement Service starts successfully however after a few seconds it just stops. The OS on our server went corrupt last week and force us to reload the…
JPVoogt
  • 223
  • 2
  • 10
2
votes
1 answer

How to ignore columns in parquet/polybase import?

I'm using polybase to import a parquet file. Over time, it is likely we may add or remove named columns in the file. When I add an additional column, I get the below error: External file access failed due to internal error: 'File test.parquet:…
Neil P
  • 1,294
  • 3
  • 20
  • 38
2
votes
0 answers

PolyBase to CSV File

I'm trying to see if it's possible to create a PolyBase connection to a .CSV file. I'm using the Microsoft Access Text Driver and have set up a system DSN to the folder where the csv file is located. I'm able to query the file using OPENROWSET and…
MrTCS
  • 709
  • 4
  • 14
2
votes
1 answer

Reading decimal mark comma with an external table

SQL Server 2016, SQL Server Management Studio (SSMS). Creating external tables with Transact-SQL and Polybase. How can we read decimals with a decimal mark, comma instead of dot? When using float and decimal, it fails and throws an error. When using…
SNR
  • 139
  • 6
1
vote
1 answer

How do I get wildcard partition pruning in mssql 2022 , polybase and a s3-storage to work

Executing an OPENROWSET query with wildcards in the bulk path and the arguments in the WHERE clause seems not to work when using PolyBase to connect to S3-compatible storage. I have an on-premises SQL Server 2022 (16.0.4120.1) with PolyBase enabled.…
Stomik
  • 11
  • 2
1
vote
0 answers

Polybase In SQL server 2019 and 2022 take so long time

I'm using polybase to load data from PostgreSQL and have some problem The PostgreSQl database and SQL server (for my case is SQL server 2022) are in same machine. I create an external table in SQL server to loading data from PostgreSQL database by…
Duc Trinh
  • 11
  • 2
1
vote
1 answer

ETL table from one SQL Server to another SQL Server

I would like to ETL a table from one instance of SQL Server to another instance of SQL Server. I do not want to create an SSIS package. I have tried to extract the data using an External Table on the destination (i.e. I use the External Table to…
xhr489
  • 827
  • 11
  • 30
1
vote
0 answers

Polybase converts getdate() to the wrong format

We have two SQL Servers (both running Enterprise version on 15.0.4083.2) where we use queries to External tables via Polybase from one server to the other. We have encountered a strange error when we use the getdate() function to filter against a…
GHauan
  • 615
  • 8
  • 24
1
2