[Cplex] Integrating OpenMP and Cilk into C++

Al Grant Al.Grant at arm.com
Tue Jun 25 11:28:24 CEST 2013


Herb Sutter writes:


Ø  And C is in a wonderful position – if you do it for C, *everyone* benefits because 99.9% of us sit on top of C-based operating systems (not just POSIX). That’s why it belongs in C.

But the language semantics, and the runtime, should support languages with a concept of exceptions – i.e. support dealing with uncaught exceptions from tasks, and address scaling issues when throwing exceptions from parallel-for loops.

I haven’t been able to track down a definitive statement of what OpenMP does with uncaught C++ exceptions (there are various blog posts complaining that it’s a problem area), but having them terminate the process seems typical.   Google ‘executors’ proposal (WG21 N3378) also has uncaught exceptions from closures terminating the process.

Cilk Plus does a better job of percolating exceptions up to the spawning task, but the description in
http://software.intel.com/sites/products/documentation/doclib/iss/2013/compiler/cpp-lin/GUID-4E7C022C-557B-4366-9F67-30AAD4C2C6E1.htm suggests there are some non-obvious issues with exception-handling regions stopping parallelism happening – they have a workaround involving a 1-iteration cilk_for loop.  I would hope this wouldn’t be necessary in any standardized language features.

Actually, exception handling might be a precedent for some of our issues.  As with parallelism, it is a case where language constructs are turned (by a compiler) into calls to a runtime library which has to do low-level things like switching from one context to another.  The C++ community could have standardized the runtime library interface but did not; instead this has been done separately for various platforms, ABIs etc. This has provided some degree of interoperability and stability on each platform, but hasn’t solved the problem of throwing exceptions between languages.

Al


From: cplex-bounces at open-std.org [mailto:cplex-bounces at open-std.org] On Behalf Of Herb Sutter
Sent: 23 June 2013 01:10
To: Tom Scogland; Kim, Wooyoung
Cc: cplex at open-std.org
Subject: Re: [Cplex] Integrating OpenMP and Cilk into C++

> In my opinion, if we do not specify a scheduler interface as part of this effort, we will have done nothing worth doing.

I know I’m repeating myself, but I just have to say: +1

(I’m not saying we shouldn’t also do more at a higher level in *addition* to a general-purpose scheduler interface, just that we shouldn’t do the former *instead* of the latter.)

And C is in a wonderful position – if you do it for C, *everyone* benefits because 99.9% of us sit on top of C-based operating systems (not just POSIX). That’s why it belongs in C.

Herb


From: tom.scogland at gmail.com [mailto:tom.scogland at gmail.com] On Behalf Of Tom Scogland
Sent: Friday, June 21, 2013 2:29 PM
To: Kim, Wooyoung
Cc: Herb Sutter; Bronis R. de Supinski; Pablo Halpern; cplex at open-std.org
Subject: Re: [Cplex] Integrating OpenMP and Cilk into C++

In my opinion, if we do not specify a scheduler interface as part of this effort, we will have done nothing worth doing.  Neither Cilk nor OpenMP are useful without a runtime managing concurrency, scheduling work and (in some fashion) allowing the user to control concurrency.  Either one can be used without one, but then all that's left is an overly verbose serial program (well, with better than average SIMD usage, but I digress).  I am personally not interested in specifying a parallel language extension with no standard way to control its behavior.

In that sense I believe we need to specify *something* as a standard scheduler interface. The question at hand is not necessarily whether we should attempt to make *a single scheduler* which is perfect for all occasions, a nigh impossible task on the best of days, but rather a standard interface and mechanism for composing schedulers within an application, probably a standard library interface of some sort.  In that fashion we allow users and runtime implementers to define schedulers which fit their needs, but still incorporate into the standard system.

As Herb mentions, composability is something we need even in pure C applications, so why not do it and then offer it as a possible broader standard once we have something working for C?


On Fri, Jun 21, 2013 at 5:35 PM, Kim, Wooyoung <wooyoung.kim at intel.com<mailto:wooyoung.kim at intel.com>> wrote:
I agree with Herb wholeheartedly that composibility gets more important and sharing is essential.
But at the same time I agree with Pablo and Clark that the composability/scheduler issue is too generic/broad to address it in this study group.
IMHO, it should be addressed in a larger context because not only C/C++ but also other languages embrace parallelism (and/or concurrency).

My 2 cent-worth thought.

best regards,

-------------------------
Wooyoung Kim
+1 217-621-9968<tel:%2B1%20217-621-9968>
wooyoung dot kim at intel dot com
-----Original Message-----
From: cplex-bounces at open-std.org<mailto:cplex-bounces at open-std.org> [mailto:cplex-bounces at open-std.org<mailto:cplex-bounces at open-std.org>] On Behalf Of Herb Sutter
Sent: Friday, June 21, 2013 7:51 AM
To: Bronis R. de Supinski; Pablo Halpern
Cc: cplex at open-std.org<mailto:cplex at open-std.org>
Subject: Re: [Cplex] Integrating OpenMP and Cilk into C++

What Bronis said.

We have SC folks on the thread, but I come at it from a different direction: the ~3 million mainstream developers using C and C++.

The mainstream of the industry is arriving at the point where having parallelism is insufficient without also having _composable_ parallelism: Having N libraries in the same application, each of which uses parallelism, and they work correctly when combined.

Back in 2005-2010 as we were bringing parallelism into the mainstream, mainstream tools didn't enable programmers to write parallel code, so the first-order problem was to enable them to express it at all. If all they could do was express it in a silo where their code expected to take over the machine, that was good enough then to get them started, so a few years ago we would rightly have said that sharing is a QoI issue.

But IMO sharing is now essential, because if it's in a Standard it has to be usable in composable libraries -- people will need and expect that. Besides code portability, one of the major points of a language Standard, and especially of C in particular since the 70s, is to enable composable libraries. That's why I said before that IMO we cannot live with a long-term requirement that applications use multiple schedulers -- including in combined C and C++ projects, but even just in multi-library C projects. So IMO we do have to think about compatibility, and because C is the substrate of the world's OS infrastructure, it's an excellent place for addressing this need.

Herb


> -----Original Message-----
> From: cplex-bounces at open-std.org<mailto:cplex-bounces at open-std.org> [mailto:cplex-bounces at open-std.org<mailto:cplex-bounces at open-std.org>]
> On Behalf Of Bronis R. de Supinski
> Sent: Thursday, June 20, 2013 6:10 PM
> To: Pablo Halpern
> Cc: cplex at open-std.org<mailto:cplex at open-std.org>
> Subject: Re: [Cplex] Integrating OpenMP and Cilk into C++
>
>
> Pablo:
>
> Re:
> > This study group is a subgroup of the C standards committee.  Since
> > Cilk Plus is not a standard, it makes sense for this group to be
> > considering which parts of Cilk Plus should be part of the C standard.
>
> That sounds rather parochial. I thought this group was considering how
> to add higher level parallelism to C.
>
> > OpenMP IS a standard, but it is not part of the C standard. As you
> > point out, it does not interoperate with Cilk Plus.  In fact, parts
> > of OpenMP do not compose well with some other parts of OpenMP.
>
> Exceuse me? Could you be specific? I disagree with you vehemently. The
> fact that OpenMP has a far more extensive user history and
> implementation base makes it a better'
> candidate for adoption.
>
> > It is not part of the charter of the C committee to dictate how
> > runtime libraries are implemented and whether two parallelism
> > systems with different origins should share a common thread pool.
> > All of that is a Quality of Implementation (QUI) issue.
>
> What was proposed is not a quality of implementation issue, it is an
> issue of semantics for high-level parallel constructs.
>
> Bronis
>
>
>
> > It makes sense for this study group to take the best ideas from Cilk
> > Plus and OpenMP and move them into the C standard as a unified thing.
> > Once that's done, the implementers can figure out how to make them
> > play nice with their implementations of Cilk Plus and OpenMP.
> >
> > BTW, Cilk Plus is NOT a threading library.  The runtime may use OS
> > threads, the concepts are parallelism, not concurrency.  The sooner
> > we stop thinking in terms of threads, the sooner we can really learn
> > to write composable and scalable parallel programs.
> >
> > -Pablo
> >
> > On 06/20/2013 10:19 AM, Hoeflinger, Jay P wrote:
> >> Thanks for clarifying.  I think my argument applies to creating
> >> yet-another-threading-model within C just as much as it applies to
> >> making one within C++.  By the way, I am speaking for myself, not
> >> Intel Corporation.
> >>
> >> Jay
> >>
> >> *From:*John Benito [mailto:benito at bluepilot.com<mailto:benito at bluepilot.com>]
> >> *Sent:* Wednesday, June 19, 2013 6:14 PM
> >> *To:* Hoeflinger, Jay P
> >> *Cc:* cplex at open-std.org<mailto:cplex at open-std.org>
> >> *Subject:* Re: [Cplex] Integrating OpenMP and Cilk into C++
> >>
> >> This study group is a WG 14 (C) study group, not a WG 21 (C++)
> >> study
> group.
> >>
> >> /John BenitoISO/IEC JTC 1/SC 22/WG 14 - Convener///
> >>
> >> On Jun 19, 2013, at 4:04 PM, "Hoeflinger, Jay P"
> >> <jay.p.hoeflinger at intel.com<mailto:jay.p.hoeflinger at intel.com> <mailto:jay.p.hoeflinger at intel.com<mailto:jay.p.hoeflinger at intel.com>>>
> wrote:
> >>
> >>
> >>
> >> I'm struggling with what it means to merge OpenMP and Cilk into C++.
> >>   OpenMP and Cilk already exist outside of C++ and can both be used
> >> today in a C++ program.  Pulling syntax or semantics of OpenMP and
> >> Cilk into C++ for some vendors would just mean a large effort to
> >> move code from one part of their compiler and/or runtime to
> >> another, with the result being no more than we already have today.
> >> And today, the complexity is less because the implementations are partitioned.
> >>
> >> The thing that we don't have today, that makes this something
> >> useful to discuss, is some way of making the OpenMP, Cilk, and C++
> >> threading models work together.  I think that should be the focus of this effort.
> >>   I have lobbied within OpenMP for better interoperability with
> >> other threading models, but nothing has come of that yet.  Perhaps
> >> *this* effort can allow that to start happening.
> >>
> >> The interoperability problem comes down to knowing how many threads
> >> to use for an OpenMP parallel region, and how to manage thread
> >> usage with Cilk.  For that, the scheduler needs to predict future thread usage.
> >>   One way to do that is to give the programmer some set of API
> >> routines to give the schedulers hints.  Other API routines could be
> >> used to query the current overall threading state, to allow
> >> programmers to adjust their parallelism accordingly.
> >>
> >> So, I say we should keep OpenMP and Cilk separate: to allow C++ to
> >> be as agile as possible, reduce complexity, and allow OpenMP and
> >> Cilk to continue changing in their own organic ways, but add ways
> >> that allow them to work together with C++ threads and each other.
> >>
> >> Jay
> >>
> >>
> >> -----Original Message-----
> >> From: cplex-bounces at open-std.org<mailto:cplex-bounces at open-std.org> <mailto:cplex-bounces at open-<mailto:cplex-bounces at open->
> std.org<http://std.org>>
> >> [mailto:cplex-bounces at open-std.org<mailto:cplex-bounces at open-std.org> <mailto:bounces at open-std.org<mailto:bounces at open-std.org>>]
> On
> >> Behalf Of cplex-request at open-std.org<mailto:cplex-request at open-std.org>
> >> <mailto:cplex-request at open-std.org<mailto:cplex-request at open-std.org>>
> >> Sent: Wednesday, June 19, 2013 4:12 PM
> >> To: cplex at open-std.org<mailto:cplex at open-std.org> <mailto:cplex at open-std.org<mailto:cplex at open-std.org>>
> >> Subject: Cplex Digest, Vol 2, Issue 19
> >>
> >> Send Cplex mailing list submissions to cplex at open-std.org<mailto:cplex at open-std.org>
> >> <mailto:cplex at open-std.org<mailto:cplex at open-std.org>>
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >> http://www.open-std.org/mailman/listinfo/cplex
> >> or, via email, send a message with subject or body 'help' to
> >> cplex-request at open-std.org<mailto:cplex-request at open-std.org> <mailto:cplex-request at open-std.org<mailto:cplex-request at open-std.org>>
> >>
> >> You can reach the person managing the list at
> >> cplex-owner at open-std.org<mailto:cplex-owner at open-std.org> <mailto:cplex-owner at open-std.org<mailto:cplex-owner at open-std.org>>
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than
> >> "Re: Contents of Cplex digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >>    1. Re: Cplex: suggested topics for discussion on the next
> >>       teleconf. (Jeffrey Yasskin)
> >>    2. Re: Cplex: suggested topics for discussion on thenext
> >>       teleconf. (Darryl Gove)
> >>
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -
> >>
> >> Message: 1
> >> Date: Wed, 19 Jun 2013 10:43:04 -0700
> >> From: Jeffrey Yasskin <jyasskin at google.com<mailto:jyasskin at google.com>
> >> <mailto:jyasskin at google.com<mailto:jyasskin at google.com>>>
> >> Subject: Re: [Cplex] Cplex: suggested topics for discussion on the
> >> nextteleconf.
> >> To: Herb Sutter <hsutter at microsoft.com<mailto:hsutter at microsoft.com>
> >> <mailto:hsutter at microsoft.com<mailto:hsutter at microsoft.com>>>
> >> Cc: Artur Laksberg <Artur.Laksberg at microsoft.com<mailto:Artur.Laksberg at microsoft.com>
> >> <mailto:Artur.Laksberg at microsoft.com<mailto:Artur.Laksberg at microsoft.com>>>,
> >> "chandlerc at google.com<mailto:chandlerc at google.com> <mailto:chandlerc at google.com<mailto:chandlerc at google.com>>"
> >> <chandlerc at google.com<mailto:chandlerc at google.com> <mailto:chandlerc at google.com<mailto:chandlerc at google.com>>>,Niklas
> >> Gustafsson <Niklas.Gustafsson at microsoft.com<mailto:Niklas.Gustafsson at microsoft.com>
> >> <mailto:Niklas.Gustafsson at microsoft.com<mailto:Niklas.Gustafsson at microsoft.com>>>,"cplex at open-std.org<mailto:cplex at open-std.org>
> >> <mailto:cplex at open-std.org<mailto:cplex at open-std.org>>"
> >> <cplex at open-std.org<mailto:cplex at open-std.org> <mailto:cplex at open-std.org<mailto:cplex at open-std.org>>>
> >> Message-ID:
> >> <CANh-
> dX=9tNk+mnpxz3WCoNHH=JLmkeJJLn1C_0+kc28EPC9Pjw at mail.gmail.com<mailto:JLmkeJJLn1C_0%2Bkc28EPC9Pjw at mail.gmail.com>
> >> <mailto:CANh-<mailto:CANh->
> dX=9tNk+mnpxz3WCoNHH=JLmkeJJLn1C_0+kc28EPC9Pjw at mail.gmai<mailto:dX=9tNk+mnpxz3WCoNHH=JLmkeJJLn1C_0+kc28EPC9Pjw at mail.gmai>
> >> l.com<http://l.com>>>
> >> Content-Type: text/plain; charset="windows-1252"
> >>
> >> On Wed, Jun 19, 2013 at 7:57 AM, Herb Sutter <hsutter at microsoft.com<mailto:hsutter at microsoft.com>
> >> <mailto:hsutter at microsoft.com<mailto:hsutter at microsoft.com>>> wrote:
> >>
> >>
> >> *[adding 4 folks to the To: line who are working on parallel
> >> ?executors?/schedulers in WG21/SG1, but I?m not sure if they?re on
> >> this list ? they may have relevant comments about the possibility
> >> of standardizing a cross-language scheduling layer at the C level]*
> >>
> >> **
> >>
> >>
> >> 2 thoughts:
> >>
> >> (0: your email's quoting is so confusing.)
> >>
> >>
> >>
> >>
> >> As Hans wrote:****
> >>
> >> I fully agree with the need for a common parallel runtime across
> >> languages. One could go even further and ask for a common runtime
> >> across applications, allowing applications to adapt their degree of
> >> parallelism to the current system load.****
> >>
> >> Yes. IMO the three problems we?re solving in the mainstream
> >> industry, in necessary order, are: (A) Make it possible to even
> >> express parallelism reliably. We?ve all been working on enabling
> >> that, and we?re partway there.
> >> Only once (A) is in place do you get to the second-order problems,
> >> which arise only after people can and do express parallelism: (B1)
> >> Make it possible for multiple libraries in the same application to
> >> use parallelism internally without conflicting/oversubscribing/etc.
> >> = common intra-app scheduler. (B2) Ditto across multiple
> >> applications, driving the scheduling into the OS (or equivalent
> >> inter-app/intra-machine scheduler).****
> >>
> >> ** **
> >>
> >> It would be immensely valuable to standardize (B).****
> >>
> >> **
> >>
> >> Establishing a common scheduling runtime is quite hard, since you
> >> have to cover both CPU-bound tasks that should be limited to
> >> 1-per-core, and IO-bound tasks that need many more scheduled at once.
> >> What existing examples of this do we have to learn from? Google's
> >> internal attempt has not been very successful, IMO. Are people
> >> happy with Microsoft's system of TaskCreationOptions passed to the
> >> global TaskScheduler? Are people happy with Grand Central
> >> Dispatch's global queue options? By "happy with", I mean, do they
> >> use these to control all concurrency in their systems, or do many
> >> of them create other threads
> manually?
> >>
> >>
> >>
> >> Yes. I would love to see us undertake to first do (2) and enable
> >> different forms of (1) as library and language extensions, then see
> >> if we can standardize (1). As someone noted, there is work on (2)
> >> being done in
> >> WG21/SG1 right now with Google?s (and collaborators?) ?executors?
> proposal.
> >> Should I see if those folks can join this group if they aren?t on
> >> it already? (CC?ing three of the people on that effort.)****
> >>
> >> **
> >>
> >>
> >> Google's "executors" get a lot of mileage by assuming that users
> >> with different constraints can instantiate different executors.
> >> That assumption conflicts with the idea that we'll have one
> >> scheduling library to coordinate across multiple processes. If we
> >> get a shared scheduling library, we'd likely wrap it in a set of
> >> executors, but it shouldn't itself be an executor: it needs too many options.
> >>
> >> HTH,
> >> Jeffrey
> >> -------------- next part -------------- An HTML attachment was
> >> scrubbed...
> >> URL:
> >> http://www.open-
> std.org/pipermail/cplex/attachments/20130619/c3a95149<http://std.org/pipermail/cplex/attachments/20130619/c3a95149>
> >> /attachment-0001.html
> >>
> >>
> >> ------------------------------
> >>
> >> Message: 2
> >> Date: Wed, 19 Jun 2013 14:11:56 -0700
> >> From: Darryl Gove <darryl.gove at oracle.com<mailto:darryl.gove at oracle.com>
> >> <mailto:darryl.gove at oracle.com<mailto:darryl.gove at oracle.com>>>
> >> Subject: Re: [Cplex] Cplex: suggested topics for discussion on the
> >> nextteleconf.
> >> Cc: "'chandlerc at google.com<mailto:chandlerc at google.com>'" <chandlerc at google.com<mailto:chandlerc at google.com>
> >> <mailto:chandlerc at google.com<mailto:chandlerc at google.com>>>,Artur Laksberg
> >> <Artur.Laksberg at microsoft.com<mailto:Artur.Laksberg at microsoft.com>
> >> <mailto:Artur.Laksberg at microsoft.com<mailto:Artur.Laksberg at microsoft.com>>>,Jeffrey Yasskin
> >> <jyasskin at google.com<mailto:jyasskin at google.com> <mailto:jyasskin at google.com<mailto:jyasskin at google.com>>>,
> >> "cplex at open-std.org<mailto:cplex at open-std.org> <mailto:cplex at open-std.org<mailto:cplex at open-std.org>>" <cplex at open-
> std.org<http://std.org>
> >> <mailto:cplex at open-std.org<mailto:cplex at open-std.org>>>,Niklas Gustafsson
> >> <Niklas.Gustafsson at microsoft.com<mailto:Niklas.Gustafsson at microsoft.com>
> >> <mailto:Niklas.Gustafsson at microsoft.com<mailto:Niklas.Gustafsson at microsoft.com>>>
> >> Message-ID: <51C21E9C.8090801 at oracle.com<mailto:51C21E9C.8090801 at oracle.com>
> >> <mailto:51C21E9C.8090801 at oracle.com<mailto:51C21E9C.8090801 at oracle.com>>>
> >> Content-Type: text/plain; charset=windows-1252; format=flowed
> >>
> >> Hi,
> >>
> >> This is an interesting discussion. I'd like to try and capture what
> >> I think the key points are, pulling in some of the earlier
> >> discussion on the alias.
> >>
> >> We have some general parallelisation concepts extracted from Cilk
> >> and OpenMP which are tasks, parallel for, parallel region, reductions, etc.
> >>
> >> In OpenMP we have a set of what might be called scheduling or
> >> execution directives which have no direct equivalent in Cilk.
> >>
> >> In Cilk we have composability because everything resolves down to
> >> tasks, and the tasks can be placed on a "single" queue and
> >> therefore it doesn't matter who produced the tasks because they all
> >> end up on the
> same queue.
> >> In OpenMP we have to manage composability through nested parallelism.
> >> This gives us more control over which threads perform a task, or
> >> where that task is executed, but it makes it difficult if you have
> >> nested parallelism from combined applications and libraries from
> >> different sources - the developer needs to more carefully manage
> >> the
> nesting.
> >>
> >> The recent discussions on this alias have talked about
> >> "schedulers", and the Ada paper talked about "parallelism manager".
> >> I've not seen a definitive definition, so I'm mapping them onto
> >> what amounts to a thread pool plus some kind of "how do I schedule
> >> the work" manager (which kind-of looks a bit like a beefed up OpenMP schedule directive).
> >>
> >> Conceptually I think we can do the following.
> >>
> >> We have a parallelism manager which has a pool of threads. Each
> >> thread could be bound to a particular hardware thread or locality
> >> group. The parallelism manager also handles how a new task is
> >> handled, and which task is picked next for execution.
> >>
> >> A parallel program has a default manager which has a single pool of
> >> threads - which would give Cilk-like behaviour. If we encounter a
> >> parallel region, or a parallel-for, the generated tasks are
> >> assigned to the default manager.
> >>
> >> However, we can also create a new manager, give it some threads,
> >> set up a scheduler, and then use that manager in a delineated region.
> >> This could enable us to provide the same degree of control as
> >> nested parallelism provides in OpenMP.
> >>
> >> For example:
> >>
> >> parallel-for(...) {...} // would use the current manager.
> >>
> >> Or
> >>
> >> p_manager_t pman = my_new_manager();
> >>
> >> p_manager_t_ old_pman = _Use_manager(pman);
> >>
> >> parallel_for(...) {...} // would use a new manager for this loop
> >>
> >> _Use_manager(old_pman);
> >>
> >> [Note: I'm not proposing this as syntax or API, just trying out the
> >> concept.]
> >>
> >> If the above doesn't seem too outlandish, then I think we can
> >> separate the "parallelism manager" from the parallelism keywords.
> >> So we should be able to put together a separate proposal for the "manager".
> >>
> >> This is good because the starting point proposal that Robert and I
> >> provided was based on existing Cilk/OpenMP functionality. This
> "manager"
> >> concept is less nailed down, so would presumably take a bit more
> refinement.
> >>
> >> One of the other comments was about how this works on a system-wide
> >> level, where multiple applications are competing for resources.
> >> That is a concern of mine as well. But reflecting on that issue
> >> this morning, it's not that dissimilar to the current situation. We
> >> will certainly be making it easier to develop applications that
> >> request multiple threads, but the instance of Thunderbird that I'm
> >> currently running has 98 threads. I rely on the OS to mediate, or I
> >> can potentially partition the system to appropriately allocate resources.
> >> Hence I'm not convinced that we need to prioritise solving this in
> >> the general case, and potentially it becomes a separate "proposal"
> >> that
> works with the "manager" proposal.
> >>
> >> Regards,
> >>
> >> Darryl.
> >>
> >>
> >> _______________________________________________
> >> Cplex mailing list
> >> Cplex at open-std.org<mailto:Cplex at open-std.org> <mailto:Cplex at open-std.org<mailto:Cplex at open-std.org>>
> >> http://www.open-std.org/mailman/listinfo/cplex
> >>
> >>
> >>
> >> _______________________________________________
> >> Cplex mailing list
> >> Cplex at open-std.org<mailto:Cplex at open-std.org>
> >> http://www.open-std.org/mailman/listinfo/cplex
> >>
> >
> > _______________________________________________
> > Cplex mailing list
> > Cplex at open-std.org<mailto:Cplex at open-std.org>
> > http://www.open-std.org/mailman/listinfo/cplex
> >
> _______________________________________________
> Cplex mailing list
> Cplex at open-std.org<mailto:Cplex at open-std.org>
> http://www.open-std.org/mailman/listinfo/cplex
>


_______________________________________________
Cplex mailing list
Cplex at open-std.org<mailto:Cplex at open-std.org>
http://www.open-std.org/mailman/listinfo/cplex
_______________________________________________
Cplex mailing list
Cplex at open-std.org<mailto:Cplex at open-std.org>
http://www.open-std.org/mailman/listinfo/cplex



--
-Tom Scogland
http://tom.scogland.com
"A little knowledge is a dangerous thing.
 So is a lot."
-Albert Einstein


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.open-std.org/pipermail/cplex/attachments/20130625/bdc3e472/attachment.html 


More information about the Cplex mailing list