Skip to content
This repository was archived by the owner on Nov 7, 2023. It is now read-only.

Commit 7bed841

Browse files
author
David Noble
committed
Release 1.8.1 addresses two bugs and updates the build system.
* On import the Cosmos DB provider now ignores Json.NET metadata. All Json.NET metadata is now passed through unchanged. This prevents failures when by chance the metadata is incorrect/unrecognized by the Json.NET serialization code. * Fixed NumOpsPerBatch and default value of NumBytestoBufferBeforeFlushing The Azure Tables provider code was not honoring the max-operations-per-batch limit of 100. Fixed this. We have a setting for the max-bytes-to-buffer before starting to split it into batches and flushing the batches. The default value for this setting was 1GB. One customer reported excessive memory usage on account of this setting. Reduced this to 10MB to fix the memory usage issue. The point to note is that when reading values from Std/Preview Tables the partitionkeys are read in a sequence -- we get all records with a given paritionkey before moving onto the next one. Therefore a lower memory buffer limit should be sufficient. * The project system now builds for x64, not Any CPU. This is a recognition of the fact that the Cosmos DB provider depends on two native 64-bit assemblies.
1 parent 43a0ff1 commit 7bed841

File tree

76 files changed

+513
-402
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+513
-402
lines changed

‎AzureTable/Microsoft.DataTransfer.AzureTable.FunctionalTests/Microsoft.DataTransfer.AzureTable.FunctionalTests.csproj‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
<Project ToolsVersion="12.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
33
<PropertyGroup>
44
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
5-
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
5+
<Platform Condition=" '$(Platform)' == '' ">x64</Platform>
66
<ProjectGuid>{C23E6A57-2C64-4924-B6C6-BAA511C9DDD3}</ProjectGuid>
77
<OutputType>Library</OutputType>
88
<AppDesignerFolder>Properties</AppDesignerFolder>
@@ -19,7 +19,7 @@
1919
<NuGetPackageImportStamp>
2020
</NuGetPackageImportStamp>
2121
</PropertyGroup>
22-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
22+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|x64' ">
2323
<DebugSymbols>true</DebugSymbols>
2424
<DebugType>full</DebugType>
2525
<Optimize>false</Optimize>
@@ -28,7 +28,7 @@
2828
<ErrorReport>prompt</ErrorReport>
2929
<WarningLevel>4</WarningLevel>
3030
</PropertyGroup>
31-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
31+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|x64' ">
3232
<DebugType>pdbonly</DebugType>
3333
<Optimize>true</Optimize>
3434
<OutputPath>bin\Release\</OutputPath>

‎AzureTable/Microsoft.DataTransfer.AzureTable.Wpf/Microsoft.DataTransfer.AzureTable.Wpf.csproj‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
44
<PropertyGroup>
55
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
6-
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
6+
<Platform Condition=" '$(Platform)' == '' ">x64</Platform>
77
<ProjectGuid>{8A191B07-775F-43C8-ACB6-F742073612EA}</ProjectGuid>
88
<OutputType>library</OutputType>
99
<AppDesignerFolder>Properties</AppDesignerFolder>
@@ -14,7 +14,7 @@
1414
<ProjectTypeGuids>{60dc8134-eba5-43b8-bcc9-bb4bc16c2548};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
1515
<WarningLevel>4</WarningLevel>
1616
</PropertyGroup>
17-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
17+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|x64' ">
1818
<DebugSymbols>true</DebugSymbols>
1919
<DebugType>full</DebugType>
2020
<Optimize>false</Optimize>
@@ -23,7 +23,7 @@
2323
<ErrorReport>prompt</ErrorReport>
2424
<WarningLevel>4</WarningLevel>
2525
</PropertyGroup>
26-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
26+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|x64' ">
2727
<DebugType>pdbonly</DebugType>
2828
<Optimize>true</Optimize>
2929
<OutputPath>bin\Release\</OutputPath>

‎AzureTable/Microsoft.DataTransfer.AzureTable/ConfigurationResources.Designer.cs‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎AzureTable/Microsoft.DataTransfer.AzureTable/ConfigurationResources.resx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
<?xml version="1.0" encoding="utf-8"?>
1+
<?xml version="1.0" encoding="utf-8"?>
22
<root>
33
<!--
44
Microsoft ResX Schema
@@ -121,7 +121,7 @@
121121
<value>Optional, default is 2MB. Specify the batch size in bytes</value>
122122
</data>
123123
<data name="BulkSink_MaxInputBufferSize" xml:space="preserve">
124-
<value>Optional, default is 1GB. Approximate estimate of input bytes to buffer before flushing data to sink</value>
124+
<value>Optional, default is 10MB. Approximate estimate of input bytes to buffer before flushing data to sink</value>
125125
<comment>This is a hint to the import tool to keep memory usage close to this limit. It is not a strictly enforced limit</comment>
126126
</data>
127127
<data name="BulkSink_Overwrite" xml:space="preserve">
@@ -151,4 +151,4 @@
151151
<data name="Source_Table" xml:space="preserve">
152152
<value>Name of the Azure Table</value>
153153
</data>
154-
</root>
154+
</root>

‎AzureTable/Microsoft.DataTransfer.AzureTable/Microsoft.DataTransfer.AzureTable.csproj‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
44
<PropertyGroup>
55
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
6-
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
6+
<Platform Condition=" '$(Platform)' == '' ">x64</Platform>
77
<ProjectGuid>{366BA489-E851-4899-9BA3-2F9C7599D24B}</ProjectGuid>
88
<OutputType>Library</OutputType>
99
<AppDesignerFolder>Properties</AppDesignerFolder>
@@ -14,7 +14,7 @@
1414
<NuGetPackageImportStamp>
1515
</NuGetPackageImportStamp>
1616
</PropertyGroup>
17-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
17+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|x64' ">
1818
<DebugSymbols>true</DebugSymbols>
1919
<DebugType>full</DebugType>
2020
<Optimize>false</Optimize>
@@ -23,7 +23,7 @@
2323
<ErrorReport>prompt</ErrorReport>
2424
<WarningLevel>4</WarningLevel>
2525
</PropertyGroup>
26-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
26+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|x64' ">
2727
<DebugType>pdbonly</DebugType>
2828
<Optimize>true</Optimize>
2929
<OutputPath>bin\Release\</OutputPath>

‎AzureTable/Microsoft.DataTransfer.AzureTable/Sink/BatchSizeTracker.cs‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
using Microsoft.Azure.CosmosDB.Table;
1+
using Microsoft.Azure.CosmosDB.Table;
22
using Microsoft.DataTransfer.TableAPI.Sink.Bulk;
33
using System.Collections.Generic;
44

@@ -11,6 +11,7 @@ public class BatchSizeTracker
1111
{
1212
private InputSizeTracker _inputSizeTracker;
1313
private long _maxBatchSizeInBytes;
14+
private const int _maxEntriesPerBatch = 100;
1415

1516
/// <summary>
1617
/// Create an instance of the batch size tracker
@@ -32,17 +33,21 @@ public IEnumerable<IEnumerable<TableOperation>> Split(IEnumerable<TableOperation
3233
{
3334
var result = new List<List<TableOperation>>();
3435
long sum = 0;
36+
int numEntries = 0;
3537

3638
foreach (var op in list)
3739
{
3840
var entity = op.Entity;
3941
long docLength = _inputSizeTracker.GetDocumentLength(entity.PartitionKey, entity.RowKey);
4042

41-
if (result.Count > 0 && (sum += docLength) <= _maxBatchSizeInBytes)
43+
if (result.Count > 0 &&
44+
(sum += docLength) <= _maxBatchSizeInBytes &&
45+
(numEntries += 1) <= _maxEntriesPerBatch)
4246
result[result.Count - 1].Add(op);
4347
else
4448
{
4549
sum = docLength;
50+
numEntries = 1;
4651
result.Add(new List<TableOperation> { op });
4752
}
4853
}

‎AzureTable/Microsoft.DataTransfer.AzureTable/Sink/TableAPIBulkSinkAdapterFactory.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
namespace Microsoft.DataTransfer.TableAPI.Sink.Bulk
1+
namespace Microsoft.DataTransfer.TableAPI.Sink.Bulk
22
{
33
using Microsoft.DataTransfer.AzureTable;
44
using Microsoft.DataTransfer.Basics;
@@ -38,7 +38,7 @@ public async Task<IDataSinkAdapter> CreateAsync(ITableAPIBulkSinkAdapterConfigur
3838
if (String.IsNullOrEmpty(configuration.TableName))
3939
throw Errors.TableNameMissing();
4040

41-
long maxInputBufferSizeInBytes = 1 * 1024 * 1024 * 1024;
41+
long maxInputBufferSizeInBytes = 10 * 1024 * 1024;
4242
if (configuration.MaxInputBufferSize.HasValue)
4343
maxInputBufferSizeInBytes = configuration.MaxInputBufferSize.Value;
4444

‎AzureTable/Microsoft.DataTransfer.AzureTable/packages.config‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
<?xml version="1.0" encoding="utf-8"?>
1+
<?xml version="1.0" encoding="utf-8"?>
22
<packages>
33
<package id="Microsoft.Azure.CosmosDB.Table" version="0.9.0-preview" targetFramework="net451" />
44
<package id="Microsoft.Azure.DocumentDB" version="1.19.0" targetFramework="net451" />
@@ -16,4 +16,5 @@
1616
<package id="System.Linq.Queryable" version="4.0.0" targetFramework="net451" />
1717
<package id="System.Net.Requests" version="4.0.11" targetFramework="net451" />
1818
<package id="System.Spatial" version="5.8.3" targetFramework="net451" />
19-
</packages>
19+
<package id="WindowsAzure.Storage-PremiumTable" version="0.1.0-preview" targetFramework="net451" />
20+
</packages>

‎Console/Microsoft.DataTransfer.ConsoleHost.DynamicConfiguration/Microsoft.DataTransfer.ConsoleHost.DynamicConfiguration.csproj‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
44
<PropertyGroup>
55
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
6-
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
6+
<Platform Condition=" '$(Platform)' == '' ">x64</Platform>
77
<ProjectGuid>{043A0CB1-09E0-4FCA-B75F-6FA25A1C9235}</ProjectGuid>
88
<OutputType>Library</OutputType>
99
<AppDesignerFolder>Properties</AppDesignerFolder>
@@ -12,7 +12,7 @@
1212
<TargetFrameworkVersion>v4.5.1</TargetFrameworkVersion>
1313
<FileAlignment>512</FileAlignment>
1414
</PropertyGroup>
15-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
15+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|x64' ">
1616
<DebugSymbols>true</DebugSymbols>
1717
<DebugType>full</DebugType>
1818
<Optimize>false</Optimize>
@@ -21,7 +21,7 @@
2121
<ErrorReport>prompt</ErrorReport>
2222
<WarningLevel>4</WarningLevel>
2323
</PropertyGroup>
24-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
24+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|x64' ">
2525
<DebugType>pdbonly</DebugType>
2626
<Optimize>true</Optimize>
2727
<OutputPath>bin\Release\</OutputPath>

‎Console/Microsoft.DataTransfer.ConsoleHost.Extensibility/Microsoft.DataTransfer.ConsoleHost.Extensibility.csproj‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
44
<PropertyGroup>
55
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
6-
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
6+
<Platform Condition=" '$(Platform)' == '' ">x64</Platform>
77
<ProjectGuid>{323766CD-3E2E-42E0-BC6E-CB7785057154}</ProjectGuid>
88
<OutputType>Library</OutputType>
99
<AppDesignerFolder>Properties</AppDesignerFolder>
@@ -12,7 +12,7 @@
1212
<TargetFrameworkVersion>v4.5.1</TargetFrameworkVersion>
1313
<FileAlignment>512</FileAlignment>
1414
</PropertyGroup>
15-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
15+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|x64' ">
1616
<DebugSymbols>true</DebugSymbols>
1717
<DebugType>full</DebugType>
1818
<Optimize>false</Optimize>
@@ -21,7 +21,7 @@
2121
<ErrorReport>prompt</ErrorReport>
2222
<WarningLevel>4</WarningLevel>
2323
</PropertyGroup>
24-
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
24+
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|x64' ">
2525
<DebugType>pdbonly</DebugType>
2626
<Optimize>true</Optimize>
2727
<OutputPath>bin\Release\</OutputPath>

0 commit comments

Comments
 (0)