REST Batching

Hi Chris,

It's a little tricky with the dataservice because it does not know about the objects you are trying to batch, so you have to use the content type of a:XElement like below:

<BatchRequest xmlns:i="http://www.w3.org/2001/XMLSchema-instance">

<Requests>

<Request>

<Content i:type="a:XElement" xmlns:a="http://schemas.datacontract.org/2004/07/System.Xml.Linq">

<CustomThing>

<CustomColumn>1</CustomColumn>

</CustomThing>

</Content>

<HttpMethod>POST</HttpMethod>

<Id>1</Id>

<Uri>https://website.com/testapi/tessituraservice/custom/customthings/</Uri>

</Request>

<Request>

<HttpMethod>GET</HttpMethod>

<Id>2</Id>

<Uri>https://website.com/testapi/tessituraservice/custom/customthings/1</Uri>

</Request>

</Requests>

</BatchRequest>

Thanks Paul-- That will probably save me some extra headaches in the future. Seems my GET call is matching yours, though, so I'm still unsure of what might be wrong with my base calls.

Do you know if the API loops back through the network to make the batch calls? I'm still not entirely sure of the stack behavior when it comes to these Batch methods

Hi Nick,

Did you say you were able to get this Batch call to work?  Tessitura has just told me that there is a defect with JSON and Batch calls that is currently under development, and have suggested I switch to XML (blech).

--Gawain

What version and hotfix are you on?

I can make this request as JSON just fine

Thanks Christopher...I'm not 100% sure, is that something I can easily look up?  The version # is 12.5.1.216.

Ah, here we go, dbo.FS_CURRENT_VERSION():

12.5.1+HF44 216

So, Hotfix 44?

That is strange I don't think that you should have a problem then.

Have you tried XML with success? What are your headers set to?

I tried to break this earlier and I was getting fairly responsive errors, it seems that the conversion was happening fine with data structure you had provided.

Hi Christopher,

I've asked Tessitura Support if this was an issue introduced by a hotfix, in case having a higher HF is what is causing me to have the problem and not you guys.

I tried XML (Tessitura Support recommended it as a workaround), but I'm having no success there either.

Here's what I'm plugging in (now almost identical to the code snipped in the documentation):

<BatchRequest>
	<Requests>
		<Request>
			<HttpMethod>POST</HttpMethod>
			<Id>1</Id>
			<Uri>http://calplb/TestAPI/TessituraService/CRM/Addresses/</Uri>
			<Content i:type="address" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
				<Constituent><Id>20335666</Id></Constituent>
				<Addresstype><Id>14</Id></Addresstype>
				<Street1>111 Nob Hill Crescent</Street1>
				<City>San Francisco</City>
				<State><Id>11</Id></State>
				<PostalCode>94109</PostalCode>
				<Country><Id>1</Id></Country>
				<Label>True</Label>
				<PrimaryIndicator>False</PrimaryIndicator>
				<Months>YYYYNNNNYYYY</Months>
			</Content>
		</Request>
	</Requests>
</BatchRequest>

And here's the error I'm getting back:

<BatchResponse xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
  <BatchFailed>true</BatchFailed>
  <Responses>
    <Response>
      <ErrorMessages>
        <ErrorMessage>
          <Code>FIELD_CANNOT_BE_NULL</Code>
          <Description>Field 'entity' cannot be null or empty.</Description>
          <Details i:nil="true" />
          <ErrorPath>entity</ErrorPath>
        </ErrorMessage>
      </ErrorMessages>
      <RequestId>1</RequestId>
      <ResponseObject i:nil="true" />
      <StatusCode>BadRequest</StatusCode>
    </Response>
  </Responses>
</BatchResponse>

Interesting.

We are on HF48 right now and are getting scheduled for 49 this week, so if that's the case it must have gotten un-borked.

Maybe update and report back if you still have any issues?

Upgraded to HF48, but no change.  Not surprised, frankly.  Something else is clearly wrong.

In the meantime, trying to work around not being able try things out in Batch, I'm realizing that I might have a bigger problem.  My assumption was that I was going to build some procedures to call through Custom/Execute, and I was going to used returned ids to fill in dependent procedures.  However, Custom/Execute isn't going to return columns in a way that I think can be referenced by the DependsOnRequest...and if did, I wouldn't be able to apply it to the dependent call to Custom/Execute since the parameters are all bunged together in a single object ("Parameters").

I'm trying to figure out TR_DATASERVICE, but documentation is even more scanty for that, particularly for formatting POST and PUT commands.

Say Christopher/Nick,

When you were running this, were you doing it through the provided Rest Client (that's what I'm doing), or via cURL or the .Net libraries?

--Gawain

I was just using the REST client provided in the services web interface.

Hi Gawain - Postman is a useful REST client. It's one I use on a regular basis: https://www.getpostman.com/.

I also used Postman to test this.

The only thing I can think of if the client is the issue would be if you're not sending the response body explicitly and there is a serialization issue with the JSON object that is being sent for the batch call [sending a serialized .NET object somehow?].

Have you tried to connect to your gateway outside of .NET at all?

Hi Christopher,

I'm trying Postman to at least rule out whether or not the built-in client is my problem, but I'm not sure I'm doing it right: I changed to bulk edit for parameters and then copied and pasted in the request data, but I'm not sure that's right.  Is there somewhere else to enter request data, or does it need to be formatted in a different way?

--Gawain

(duplicate)

Sorry, nevermind, found it.  Now I get a different error at least!

(Update: wrong content-type)

Okay, so now it just fails silently.  That's something, I guess?

Is your accept header set?

Is your accept header set?

It wasn't, and now I get the same error I get through the built-in REST client.  Oh well.  Oddly, I didn't need it to run some other (not Batch) methods correctly.

Gawain, hoping this helps a bit:

in your batch request above, try moving Content to the first element.  Using XML, the elements need to be alphabetically ordered in your request.  It's not seeing Content first and deserializing it as NULL.

<BatchRequest>
	<Requests>
		<Request>
			<HttpMethod>POST</HttpMethod>
			<Id>1</Id>
			<Uri>http://calplb/TestAPI/TessituraService/CRM/Addresses/</Uri>
			<Content i:type="address" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
				<Constituent><Id>20335666</Id></Constituent>
				<Addresstype><Id>14</Id></Addresstype>
				<Street1>111 Nob Hill Crescent</Street1>
				<City>San Francisco</City>
				<State><Id>11</Id></State>
				<PostalCode>94109</PostalCode>
				<Country><Id>1</Id></Country>
				<Label>True</Label>
				<PrimaryIndicator>False</PrimaryIndicator>
				<Months>YYYYNNNNYYYY</Months>
			</Content>
		</Request>
	</Requests>
</BatchRequest>

I have one final note regarding custom and batching.  At this time, the Custom resources are not supported in Batch requests.  The Batch service relies heavily on known service entities (like 'Constituent', 'PriceType', e.g.) to build its requests and responses.  Custom resources are by their nature ad hoc and dynamic and the capabiliity of building requests based on these does not exist in the service.

Hope to see you at TLCC!

-Ryan Creps

Tessitura Network

Hi Ryan,

Thanks!  I realized I have read that before, but totally forgot when setting this up.  That does have me moving forward a bit.  But do you mean that even Custom resources defined in TR_DATASERVICE_TABLES cannot be used with Batch?  Will that change in v14?  If that's true, I'm really sunk.

This has moved me forward with my Batch->Addresses test, but now I'm getting an error "Field 'Address Months' cannot be null or empty."  I'm specifying it, as you can see in the example, and that label is what is used in the REST documentation.  Is there some other?

Actually, running a direct call to CRM/Addresses with XML nets more errors:

<ErrorMessages xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
    <ErrorMessage>
        <Code>FIELD_CANNOT_BE_NULL</Code>
        <Description>Field 'Address Months' cannot be null or empty.</Description>
        <Details i:nil="true" />
        <ErrorPath>Address Months</ErrorPath>
    </ErrorMessage>
    <ErrorMessage>
        <Code>FIELD_CANNOT_BE_NULL</Code>
        <Description>Field 'Address City' cannot be null or empty.</Description>
        <Details i:nil="true" />
        <ErrorPath>Address City</ErrorPath>
    </ErrorMessage>
    <ErrorMessage>
        <Code>FIELD_CANNOT_BE_NULL</Code>
        <Description>Field 'Address State' cannot be null or empty.</Description>
        <Details i:nil="true" />
        <ErrorPath>Address State</ErrorPath>
    </ErrorMessage>
    <ErrorMessage>
        <Code>FIELD_CANNOT_BE_NULL</Code>
        <Description>Field 'Address PostalCode' cannot be null or empty.</Description>
        <Details i:nil="true" />
        <ErrorPath>Address PostalCode</ErrorPath>
    </ErrorMessage>
</ErrorMessages>

Still, this does still raise the question of why other people can do it (Batch->CRM/Addresses->JSON) and I can't.  Nick, Chris, one other variable (I guess), is that I'm on RAMP, and that means we are using SQL Server 2008.  I'm being told this issue is somehow fixed in v14 (which we can't have because we're stuck on SS 2008 with RAMP.  Is there a possibility that you guys are already on SS 2012?

Lastly, alas, I'm not going to be able to make the TLCC this year.  I'll miss you guys!

Huh, if I just run CRM/Addresses, and I sort the elements alphabetically:

<Address  xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
	<Addresstype><Id>14</Id></Addresstype>
	<City>San Francisco</City>
	<Constituent><Id>20335666</Id></Constituent>
	<Country><Id>1</Id></Country>
	<Label>True</Label>
	<Months>YYYYNNNNYYYY</Months>
	<PostalCode>94109</PostalCode>
	<PrimaryIndicator>False</PrimaryIndicator>
	<State><Id>11</Id></State>
	<Street1>111 Nob Hill Crescent</Street1>
</Address>

then I get a different error:

<ErrorMessages xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
    <ErrorMessage>
        <Code>FIELD_CANNOT_BE_NULL</Code>
        <Description>Field 'entity' cannot be null or empty.</Description>
        <Details>   at Tessitura.Services.Common.Guard.ArgumentNotNull(Expression`1 expression, String errorPath)
   at Tessitura.Services.Common.Guard.EntityIdShouldNotBeGreaterThanZero(IEntity entity)
   at Tessitura.Service.Impl.SubEntityService.CreateSubEntity[T](T subEntity)
   at Tessitura.Service.Impl.AddressService.Create(AddressEntity subEntity)
   at Tessitura.Service.Web.Controllers.CRM.AddressesController.&lt;&gt;c__DisplayClass7.&lt;Post&gt;b__6()
   at Tessitura.Service.Web.Controllers.Base.BaseController.RunInTransaction[T](Func`1 func)
   at Tessitura.Service.Web.Controllers.CRM.AddressesController.Post(Address address)
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.&lt;&gt;c__DisplayClass10.&lt;GetExecutor&gt;b__9(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.&lt;InvokeActionAsyncCore&gt;d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.&lt;CallOnActionExecutedAsync&gt;d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Filters.ActionFilterAttribute.&lt;CallOnActionExecutedAsync&gt;d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.&lt;ExecuteActionFilterAsyncCore&gt;d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.&lt;ExecuteAsync&gt;d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.&lt;ExecuteAuthorizationFilterAsyncCore&gt;d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ExceptionFilterResult.&lt;ExecuteAsync&gt;d__0.MoveNext()</Details>
        <ErrorPath>entity</ErrorPath>
    </ErrorMessage>
</ErrorMessages>

You are missing a few properties.  Try something along the lines of:

<Address xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
  <AddressType>
    <Id>1</Id>
  </AddressType>
  <AffiliatedConstituent/>
  <AltSalutationType>
    <Id>0</Id>
  </AltSalutationType>
  <City>San Francisco</City>
  <Constituent>
    <Id>1007</Id>
  </Constituent>
  <Country>
    <Id>1</Id>
  </Country>
  <GeoArea>1</GeoArea>
  <Id>-999</Id>
  <Inactive>false</Inactive>
  <Label>true</Label>
  <Months>YYYYYYYYYYYY</Months>
  <PostalCode>94109</PostalCode>
  <PrimaryIndicator>true</PrimaryIndicator>
  <State>
    <Id>11</Id>
  </State>
  <Street1>sample string 9</Street1>
</Address>

Bear in mind that Batch is just a convenient way of bundling calls into a single request.  This shouldn't preclude you from making requests sequentially and doing the dependencies between them yourself from each response.

There is no plan at this point to include Custom into the Batch resource.

Sorry to see that you'll be missing TLCC this year!

-Ryan

You are missing a few properties.  Try something along the lines of:

Do I really need those other fields?  I don't with JSON.  And should I even be sending "Id" on a create?  In any event, I tried this and got the same failure:

<Address  xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
	<Addresstype><Id>14</Id></Addresstype>
	<AffiliatedConstituent/>
	<AltSalutationType><Id>0</Id></AltSalutationType>
	<City>San Francisco</City>
	<Constituent><Id>20335666</Id></Constituent>
	<Country><Id>1</Id></Country>
	<GeoArea>1</GeoArea>
	<Id>-999</Id>
	<Inactive>False</Inactive>
	<Label>True</Label>
	<Months>YYYYNNNNYYYY</Months>
	<PostalCode>94109</PostalCode>
	<PrimaryIndicator>False</PrimaryIndicator>
	<State><Id>11</Id></State>
	<Street1>111 Nob Hill Crescent</Street1>
</Address>

Ryan-

Batching provides the ability to roll back if there is a failure, something that is not currently availble, or even possible, via sequential calls. That's what the docs say, at least. I haven't used it in practice yet.

For what it is worth two others in this thread made the same call with the same request body that was provided. It seems that there are some other issues at play here that don't seem to add up.

There is no plan at this point to include Custom into the Batch resource.

Man, that's a shame.

Bear in mind that Batch is just a convenient way of bundling calls into a single request.  This shouldn't preclude you from making requests sequentially and doing the dependencies between them yourself from each response.

Well, I'm building this for a partner to use to share data with us.  What happens is that we are exporting contribution data to them (they happen to be the system of record for campus contributions).  That part was simple enough, using Custom/Execute on a procedure I used to pull together all the information they needed.

Then they return some information that they file about each gift, pledge or pledge payment.  For each of those they are also going to return some of their biographical info that we lack for each relevant donor (this can include A1, A2 and any/all soft credit donors).  Lastly, for each donor, they return what they know of their education history.  So for each gift, pledge, or pledge payment entered on a given day, they return:

1. Contribution Data
   1. Donor 1 Data
      1. Degree 1
   2. Donor 2 Data
      1. Degree 1
      2. Degree 2

For X (probably about 3-50) contributions.

This I intend to have them push into three staging tables that I can then review in order to decide how to import that information into our data structures.  The ideal would have been to be able to use Batch to A) isolate a single set of returns such that on error the whole set is backed out and B) be able to use DependsOnData in order to feed row ids from one table to the next (i.e. Contribution Id -> Donor, Donor Id -> Degree), and C) that way our partner simply takes their data set and translates it into a single (ideally JSON) formatted request, and then pushes that to our API.

Without Batch they will have to build a system which pushes a row, takes an id, formats another row, sends that, takes another id, etc. and all of the batch handling, should something fail part way, falls on me.

As to TLCC, got a very high maintenance 11-month-old.  Hopefully next year!  You should hit Tucson again...

 --Gawain

Christopher,

What version of SQL Server are you guys on?

--Gawain

So, one key point to understand with Batch requests, is that each request within a batch runs in its own transaction.  What this means is if the first one succeeds, but the second one fails, the first request has been committed.

From my interpretation of this, batch wouldn't help you in this scenario because part A and B may succeed...but if part C fails, there is no rollback for requests A and B.

So the docs are wrong, ok.

So, one key point to understand with Batch requests, is that each request within a batch runs in its own transaction.  What this means is if the first one succeeds, but the second one fails, the first request has been committed.

This is the documentation that I downloaded recently:

There is no problem with making these calls independantly but there are also times when it is beneficial to group operations together. Grouping the operations may allow you to develop classes that combine commonly used operations. Also batching the operations allows them to operate as a single unit of work. If any update operation fails, all of the operations will be rolled back.

I thought you used "ContinueOnError" (In the sample request on the service pages, but not in the documentation) to toggle behaviors.

Simply put, this documentation never got updated, and I'm sorry if you were misled here. We are correcting this.

In the interim, let me clarify:

A Batch request is simply a way of bundling multiple requests.  This can be used for bundling common workflows into one request, which optimizes by only making one round trip to the service, and also saves on the cost of transport and serialization.  

Each request within a batch runs in its own transaction which will be rolled back (just like a normal request) in the event of an error.  The ContinueOnError property of the BatchRequest entity allows the batch to continue or halt in the event that one request in the sequence fails.  The default is NOT to continue on error.  Previous successful requests within the batch are not rolled back.

This change was made prior to v12 in large part due to plugin (or interceptor) behavior on requests inside the batch.  If plugins were run inside the transaction when included in a batch, the plugin would incur different behavior for the same request when run inside vs outside the batch.  To avoid this confusion and potential deadlock issues with large batch requests, we decided to remove a container transaction for the entire batch.

I hope that clarifies the intent here and we will be fixing the documentation to explain this.

Best,

Ryan Creps

I see.

Thanks so much for clearing this up and explaining it all Ryan!  This really helped me proceed with my project.

--Gawain